I heavily dislike Roger Federer, but your assessment is accurate. No doubt that he is the greatest this sport has ever seen and likely will ever see, at least for the foreseeable future. But come on, Novak ain't no slouch either. To take five slams, and possibly more, off of two of the greatest is no feat to brush off.
Nobody is saying Djokovic has done badly. Far from it. However, to be GOAT your career does not merely have to be excellent, it has to be better than anyone elses. On merits, Djokovic is sufficiently far away from being better than every other tennis player that ever lived to deserve to be ignored in the GOAT debate. If he does well enough in the coming years that will change. An additional 12 slams, 9 masters, 5 WTFs, 4 YE #1s and 250 weeks as #1 might
be enough, if he can back it up with something Federer didn't do as well.