I suppose the only way for Federer to lose is when he plays very poorly. :P
Maybe it's tied to the type of tennis he plays. He's a player that goes all out, extremely agressive style. So usually when a player follows this strategy the outcome of the match tends to be more linked to his performance rather than the opponent's.
Specially on Fed's case, he's a guy who needs to be in control and dictate the rythm. Or quickly break the opponent's rythmand adjust it to his own, you can see how easily he can accelerate a ball exchange out of the blue. Peak Novak excelled at this as well. The problems appear when he can't dictate himself, then he performs poorly since he's not much of a defender.
Just look at the Nadal matchup, it was a nightmare for Fed. Once Rafa's defense and moonball spam declined on fast courts, boom 4-0 for the old man.