Federer would have struggled in previous clay eras simply because the topspin clay court expert has always bothered him on the surface- Nadal causing him trouble wasn't the first time. Guga, Mantilla, even a past it Costa have ground him into the mud.
But that was pre-prime Fed. It took some time for him to reach his best level on clay. He reached his first RG final in 2006. That Federer wouldn't lose to a lot of players on clay even if it was his worst surface.