Disclaimer: I only think this is true, correct me if I'm wrong.

GPUs do floating-point math, in their own precision. They have way more computing cores than usual CPUs do, because they do only FP, so are smaller. Also they have their own memory on the board, which has faster access timing than RAM.

Usually we put multiple matrices (multi-dimensional array of numbers) inside, and expect some back, while the entire load of math is done on the GPU. Now it can do milions of computations very fast, in parallel, never care about anything external like RAM or goodness forbid disk or network (which is monumentally slower).

Now CUDA is the 'platform', that lets you write the code for general programming and to utilize not only the computing cores on the GPU, but also move data between the GPU, CPU and RAM.

This is a weak example, but back in the day, I mined BTC on my PC. IIRC, it just runs md5 hashes until it finds specific output. MD5 is just a mathematical algorithm, and you can run it on both CPU and GPU.

My CPU at the time (I think it was 6 core Phenom II?) could output 12 milion of md5 hashes per second. My GPU - AMD Radeon 6990 - after some tweaks and full table fan blowing from the side inside chasis could get close over 800 Mhash/s.

So there are direct incentives to use GPU for other cases than gaming, specifically machine learning is all about floating point math. But to do that, you want to be able to write your own software that implements the algorithms to squeeze every last bit of performance out of it, and that's what CUDA lets you do.

CUDA is specific to nVidia GPUs, AMD is trying to catch up with ROCm, but came almost a decade later, so they have a lot of catching up to do. Intel also started their own oneAPI, and both oneAPI and ROCm are open-source, with CUDA being closed source, so only nVidia can modify it.