I seem to recall him having difficulty with Roddick as well as Isner's recent upset.
For someone with the best return of serve of all time, I would not expect him to struggle.
Perhaps Djokovic returning Nadal's WTA serve made people think his return is better than it is?
Or is he just a big 2nd serve returner like Agassi?
Because Isner hit around 70% first serves in against Djokovic. In the first set Isner had an extreme percentage of 76%. It is even for a great returner difficult to return Isners's first serve. If Djokovic plays Isner more often I think he would get a better reading of the serve. But he hadn't play Isner since 2010.
Isner is a difficult player for everyone to play against. Especially if he plays over 70% 1st serves in. He also beat Federer a couple of weeks ago and did well agaisnt Nadal at the French Open 2011.