No, it is not silly. The umpire can not be expected to know when the net is touched, relative in time to other goings on while the ball is in play. Thus the only sensible rule is either that you can touch the net, or you can't touch the net during play of a point. Clearly, the second option could result in a player holding the net up when an opponent is returning the ball, or a player sitting on the net while his doubles partner hits the ball.
Accidently touching the bottom of the net with your foot shouldnt cost you the point.
I think the rule should be limited to touching the net cord. Any deliberate manipulation of the net should of course also be banned but that only affects Mansour Bahrani anyway.