Mathematics isn't merely about numbers, nor should it be ignored at its more complicated standard because it contains tricky concepts.
I'll say two things. Firstly, mathematics is a science, it forms an integral part of the basis of our society. None of the engineering we do, the scientific innovation, the research, the administration etc. would be possible if it weren't for mathematics. If you are truly proposing we give students the option NOT to study this essential subject, then I don't know what to say except that mathematics is behind pretty much everything, contrary to what you may think.
Secondly, it's also an acid test more than anything else. If you can succeed at GCSE mathematics, then kudos. Giving people the option to try and sneak around it would be wrong. They might as well not go to school at all.
I guess you're right, I wasn't looking that deeply into things, but Maths is every where I suppose, even though I despise it