I'm not going to say anything further as really it just must hurt all the Fed fans that no one in the tennis media cares about Fed's double medal - he's not got a career golden slam and that's that. Whatever he wins in doubles means nothing - it's the singles career that matters. Agassi and Nadal are the only players to have career golden slams - Fed, one of the greatest ever, has one little hole still on his CV and it won't be going away barring some Rio miracle...
If I would have to classify one group of fans as hurt, Fed fans would be fighting for last position in that list along with Murray fans.
Edit: You mention Agassi, pretty much every time people are talking about him and for example comparing him to Agassi, they talk about the career slam. I don't think I've ever in these debates hear anyone mention Golden career slam specifically, that tells you something on the value of it.