You're a correct. I'm a Delpo, Tsonga and Nishikori (among others) fan, so we have a lot in common as tennis fans
To answer your question, players with the variety (especially slice) to mess up his baseline rhythm - Federer is the prime example of this.
Also, players who are great at absorbing/redirecting power. Delpo's default game is to hit the ball hard with big margins to draw errors/short balls. Against 99% of the tour, as soon as he gets a full swing on a FH the rally is basically over as they can't handle the unparalleled pace/weight of shot. Against players who are great at absorbing and redirecting power though - Murray, Federer, Ferrer... - this default gameplan doesn't work at all, as they can return a lot of his FHs with decent depth. Against those players, he has to take a more aggressive approach, like he did vs Federer at Olympics for instance, they don't let him play his usual game. He needs to aim for lines and corners far more than he usually does.
Delpo hits far harder than anyone else on tour, especially off the FH, which makes him a bad matchup for the majority of the tour. Players who don't let him set up into a rhythm and those who can absorb/redirect power to great effect are his worst matchups, forcing him to change his default style to have a chance to win. Federer meets both conditions, hence why he's the worst matchup on tour for Delpo.