There aren't too many. It's not like this really slows the game down. Players take a good 20-30 seconds between points anyway. Besides, you know you just love hearing the crowd go "ooooOOOOOaaaaaaAAAAAHHHHHHH."
Challenges are useless anyway, it's just the crowd factor. If it was meant to be serious, then the umpire would have a little screen with the hawkeye results all the time, and would overturn calls automatically, without anyone asking anything.
The fact that players must do that is obviously meant just as a bit of extra fun for the public. So, 3 is just fine. 2 is too few, because once you make a wrong call, most of the time you would be reluctant to waste another, so it's basically 1 per set. More than 3 I feel would really be too many.