Exactly. The player complained because at that point he felt he was being negatively affected by someone violating the rules. That works for me and it works for the vast majority of players, I reckon.
This seems to be the point we disagree on the most. Agassi found a humorous way to indicate that it is way overboard. Most players won't react at all, but it doesn't mean it's not bothering them. It shouldn't be up to players to complain, it's the umpire's job.
It's not as if they have a clock and 1-2 seconds can be noted, an alarm would go off and a warning issued. It's about good judgment that would prevent such things as a regular occurrence. In my view, it would be far more efficient and less controversial to enforce it regularly, rather than disregard the issue for the most time and than issue one warning in an important match on an important point.