For pure explosive speed, Nadal is better than Noserer IMO. Peak Nadal's defense > peak Noserer's defense. Noserer may have slightly better anticipation, though.
As for FH, the very reason it's the best is the one you mentioned. It doesn't break down when he is defending. Noserer actually makes quite a few FH errors when in prolonged FH rallies. Nadal's DTL FH and passes are also superior.
Yup, Nadal is faster, but Federer has better footwork off clay imo, he hardly ever hit any shot out of position and, especially in his prime, could turn defense into offense against the biggest hitters seamlessly due to his insane footwork.
As for the FH, I agree Nadal's FH has a very good case for being the best FH I've seen: it's a shot that makes him a bad matchup to 99% of the tour who can't handle the wicked spin. But again, I'd hail it at the best ever on clay; on other surfaces, he can be outhit off the FH wing very often (by guys like Gonzalez, Blake, Delpo...).