The simplest way to get an LLM to "do" maths is to have it translate human language tokens relative to Maths to a standard set of Maths tokens, then passing it to a perfectly normal library that does Maths and then translating the results back into human language tokens: easy-peasy LLM "does Maths" only it doesn't, it's just integrated with something else (which was coded by a human) that does the maths and only serves as a translation layer.

Further, the actually implementation of the LLM itself is already doing Maths. For example a single neuron can add 2 numbers by having 2 inputs each with a weight of 1 and a single output because that's exactly how the simplest of neurons already calculate an output from its inputs in a standard neural networks implementation - it can do simple Maths because the very implementation is already doing maths: the "ability" to do maths is supported by the programming language in which the LLM was then coded, so the LLM would be doing maths with as much cognition as a human does food digestion.

Given the amount of bullshit in the AI domain, I would be very very weary of presuming this breakthrough being anywhere near an entirelly independent self-assembled (as in, trained rather than coded) maths engine.