I think NeHe might still have tutorials on this, in C/C++. You probably want to be using OpenGL for acceleration and maybe the old fashioned immediate mode/fixed function stuff where you call functions like "We are drawing triangles now" and "here is a vertex" and "that vertex is blue", and you can put off taking over the pipeline with your own shader code until later.

You still might be letting the library/gpu do most of "the maths" because I think you mostly hand it transformation matrices and points and it sends them to screen space. If you take over the vertex shader then your shader code does that.

You want to write a vector and matrix math library with 3-vectors and 4-vectors and 3x3 and 4x4 matrices, and add and multiply operations, and matrix inversion. The 4th dimension lets you make translation in 3D a linear multiply operation because you keep 1 in there and your matrix to represent a translation mixes that 1 into the other position coordinates to translate.

You also probably want to learn linear algebra enough for that to make sense.

And then on top of that you want to build a scene graph library where objects have parents they move with. And then your renderer loop will walk the scene graph node tree and push each object's transformation matrix and draw it and do its children and then pop the transformation matrix off again.