My daughter hates numbers and anything to do with Math so much that she simply gives up on even the simplest of sums and that too right at the very beginning itself. To some point, I can understand how she feels since I had a sort of a math-phobia myself when I was in elementary school and had this sort of firm conviction in myself that Math was simply
not meant for me, it was beyond me. Thankfully, I had this super-awesome teacher in middle school who just turned things around and I started loving the subject.
When my daughter begins to get restless while solving math, on the pretext of giving her a break, I call her into the kitchen and give her her favorite cookie to munch on. And then, our special 'cookie game' starts - wherein she needs to count how many cookies are remaining in the jar each time she takes out one (I make sure the jar is transparent and small so that the cookies can be counted by looking at them from outside). Also, whenever I am in the process of baking cookies, she loves counting how many there are in the tray, back and forth (that is primarily how I started teaching her the multiplication tables).
Your son is pretty old for these types of activities but I'm pretty sure there would be some fun activities in math you could use to get him to enjoy doing stuff. If that doesn't work out, here's a way you could make do with his short attention span; give him a cookie/chocolate treat every time he does the stipulated number of problems correctly, so he knows he is going to get a treat from mommy every time. I'm sure you'll have a better time and he'll do his sums at double the speed!