Those were in fact the only two gold medals they got last time round.
In 2004 the Swiss won only one gold medal, Marcel Fischer won the men's individual epee (fencing). He is retired now.
In 2000, likewise, only one. Birgitte McMahon won triathlon. She has long since retired.
So the normal choice when Federer and Cancellara abstains is Wawrinka.
Switzerland is not really a great Summer Olympic nation, they do better at the Winter Olympics.
Slightly off topic but what happens when a nation doesn't have an olympic medalist ? Who carries the flag then ?