If you already know some programming languages, look for some kind of GUI or game library for it to see if you can use it. If not, something like Blender might be easiest to make in C++, Rust, C (if you're a masochist), or maybe Zig. This may also influence the shading language you choose. Start with this.

You will need to know some shader language. You have a few options there, but the most popular are GLSL and OpenGL (though I'd prefer GLSL). There's also WGSL and some others, but they aren't as popular. Prefer whatever the graphics library you're using wants you to use.

Math is very heavy on linear algebra. Look up PBR if you want to render realistic 3d shapes. Google's Filament is well documented and walks through implementing it yourself if you want, but it's pretty advanced, so you might want to start simpler (fragment colors can just be base color * light color * light attenuation * (N*L) for example).