Disclaimer: we still have pragmatic reason to follow the evidence suggested by our best scientific theories. I'm just poking fun at scientists in the spirit of Hume. There's no guarantee that the future will resemble the past, and even our best scientific theories are amenable to future evidence.
Or at least our conception of reality. But I think you've hit the nail on the head here. Math is so reliable that it's unthinkable that it could ever change or lead us astray; and so most of us don't think about it at all. But (and I can already feel eyes rolling) what are numbers? Why are they universally and necessarily true? Are we inventing new mathematical truths or discovering them? These are not questions for science.
Numbers are just an abstraction for physical properties. Aliens with no contact with earth will have no idea what our Arabic numerals mean, they won't recognize a 3 or a 7 as being numbers. But they will surely know what 3 means, as a trio, that it is one and one and one.
When we get into things that have units, like the speed of light for instance, things get a little muddy because we have no concise and effective way to define our units to an outside perspective. Sure, c may be equal to around 300 million meters per second, but what is a meter and what is a second to someone who has no reference to either? It's at this point that I think math becomes more of a reflection of ourselves. We define things in units and with relations to other objects and other units because we have that frame of reference, and by defining those units and using them in relation to other maths we've created this grand interconnecting web of mathematical axioms and theories and proofs and units that all refer to four other theories and axioms and units in their own definitions. But at the base of that web holding it all together are numerals. Two is two no matter what it is two of, and that is an illustration of physical properties of the universe irrespective of units or home culture. Two is two is two even if it's called dos or İki or दो.
I would argue that we, humanity, do not invent math any more than we invented gravity or atomic weights. We merely discover principles and relations between principles and define them to the best of our understanding.
They aren't universally true because they are just names for things that are universal like quantity. when you have one item you can make any name for the word one but however it is defined it will be one and and if that where not true it would not be simply our concept of reality it would be our reality or our perception of reality which amounts to the same thing. If I find one day that I see four lights but there actually are five then either reality has changed or I have to wonder if I am mad. Or some cardasian is trying to drive me mad.