It's called knowing you can execute shots better than you currently are during rallies. He doesn't care who is opponent is, he wants to play his best, and when he has the right shot lined up, but doesn't execute it properly, he gets mad at himself. He's one of the nicest guys, but also the most critical of himself.
We should be appreciative of Federer though, because we will never see anything like him again. The fact he can still compete for slams ever after the erosion of his physical skills is really a greater testament to the natural talent he has than the period of time when he was dominating the tour.
"I don't give a rat's ass whether I'm playing a big or a small tournament. I'm Matwé Middelkoop, I go on that court and it doesn't matter who I'm facing. [...] Whether it's a blue court in Australia or a clay court in Morocco, I don't care."