Well 2008 was probably where he started to decline from his unbelievably high peak as guys like Fish, Simon, Karlovic and even Blake beat him.
The first signs I saw that he wasn't the player of 04-07 were at the 09 USO final as Fed used to eat big hitters/ballbashers alive yet he was shanking Delpo's missiles.
I think that 09 final was an anomaly, since he generally played at a very high level that season.
Federer's decline has been slower than some people like to admit. Two factors led to Federer being less successful: the players improved and the surfaces became slower. Any decline in Federer's game was relatively small until much more recently.
People were far too quick to claim that Federer was past his prime. That kind of talk began as early as the first half of 2007, when he lost to Canas. It was absurd.