Title question first: yes, you absolutely can be too dumb to program.

But as others have mentioned, being bad at math isn't necessarily a deal breaker, especially if you're taking about the actual arithmetic part of math.

What turns out to be key to programming is breaking down a problem into steps and figuring out the logic to do what you want to do. The computer is going to do the actual arithmetic, but you'll need to tell it what you want to do step by step.