In most H2Hs I've observed that the ability of the players to read the opponent's serve plays a huge role, huger than the aspects of the rallies. It's really strange but this ability varies a lot : it's far from being as easy as to say "X is a great server, Y is a poor returner then ...", it really depends who is against who.
For Fed against Ferrer, Söderling and Davydenko (also Del Potro), his alternance with the sliced backhand also plays an important role.
Add to that the fact that Ferrer can't return the Federer serve if his life depended on it and you have the most one sided head-to-head in top level tennis, only likely to become even more one sided.
I'm a Fedfan but I disagree with the final part of your comment.
7 out of 13 matches between Fed and Ferrer were until the beginning of 2007 when Ferrer was not at his best yet and Fed was in his prime.
Since then, the 2 matches they played on clay were both in Madrid, which is Fed's best clay tournament, esp. this year with the blue clay (the previous one was in 2010 and was actually tight).
Ferrer is too good on clay for Fed to feel that comfortable against him. 3-1 against Djokovic on clay (and Djoko struggled in his only win in Madrid 2011), come on ...
The other 4 matches they played after Hamburg 2007 were for 3 of them in the WTF (indoors) and Cincy, Fed's favorite tournaments.
Besides, Ferrer has won 4 (and not 3 as you said) sets in the 7 last matches they played. Ferrer has improved (notably the DTL backhands) and also he has found better tactics against Fed in their last matches, especially the forehand DTL moonballs to his backhand (followed by a crosscourt forehand sometimes).
Then no, I disagree that this H2H can only get worse, I rather think this H2H is too favourable for Fed at the moment.
Del Potro vs Nishikori
4-0 Del Potro, 10-0 in sets, most of them blowouts. Complete ownage.
-Del Potro's power/foreand. Kei likes (and manages) to dictate from the baseline even against huge hitters like Tsonga and Berdych, but he can't deal with Delpo's power and all. Especially JMDP's forehand that constantly pushes him far behind the baseline. Despite his quickness around the court, Kei lacks world class defensive skills and really struggles when he's forced to play defense, which he invariably is against Delpo. His lack of a serve allows JMDP to have the initiative even on his serve games and break him time and again (20 breaks in 10 sets). Will be tough for Kei to even win a set in this matchup in the future if he can't come up with a different strategy, rallying with Del Potro is a losing battle as the Argentine easily overpowers him.
Nishikori leads 3-1 against Berdych, 2-0 against Tsonga, I don't think it's a matter of power.
People usually mix them all under the name of "big hitters" but imo, and I observed it makes a big difference in the way the H2Hs are with many players, Del Po and Söderling are very different from Tsonga and Berdych, in the fact that they make fewer errors and their game is based on constant power and depth (you talked a little bit about depth about Del Po and that's very important) rather than accelerations like Tsonga and Berdych. Besides, Del Po and Söderling use some spin whereas Tsonga and especially Berdych (Cilic as well) play much flatter.
Besides, contrary to you, I think Nishikori has the ability to defeat Del Po in the future, just needs to adapt his strategy, I saw their match in Wimbledon and it was far from easy for Del Po and the commentator also said that Nishikori didn't use the right strategy.
Djokovic vs Cilic
7-0 Djokovic, one set for Cilic with 9-7 in the tiebreak. All very straightforward wins for Novak.
-Cilic's serve is horrible for a guy his size and a liability in general (not that strong 1st serve, poor %), Djokovic is the best returner on tour... you do the Math.
-Nole can easily frustrate Marin with his defense, making him go for too much and leak errors. Nole can even overpower him at times as well. There's basically no safe place to go for Marin in this matchup.
Cilic doesn't like great defenders and playaers who have good depth, I also remember it was his main problem against Del Po.