Everything in the physical world is physics. (I'm just now realizing how related those words are...). But mathematics can also describe purely abstract concepts, which I don't see how those are physics. It may be possible to make connections between some abstract concepts and our physical world, but it's a strong and imo incorrect claim to make that that's always meaningfully true (or will be in the future) and that that means those abstract concepts are physics.

Edit: It's certainly semantically incorrect whether there is a connection between all abstract math concepts and physics or not, because a universe with different physics would have the same math, which means physics cannot be the source of mathematical truth.