That's harsh mate, what a prick
I'm sorry but I have to laugh, and reminds me so much of the movie in this clip:
I don't think I've ever had an opponent quite as bad as the one you describe so I'm not sure I can be much help. It sounds like it was all a deliberate plan to throw you off your game, so take heart from the fact he rates you highly enough to believe the only way to beat you is by resorting to dirty tactics.
For your next match I think it really comes down to two choices:
1) Sorry to sound harsh here, but man up! Tell him you like to play tennis in good spirit and you feel his taking time between points, answering his phone, etc. is completely against the way the game should be played. You don't want to make it too serious, but blatently, he was taking the piss so call him out on it.
2) You let him continue with his antics but use it to your advantage. So he's taking extra time between points....use this to give yourself a breather, don't think about what's he's doing think about how you're going to execute the next point. Just keep positive and motivated. You are the better player so just let your tennis do the talking.
Good luck for tomorrow