Which language do i code it?

doesn't matter. If you wanna go far, pick the one you're best at. That way you have one less hurdle.

Which libraries would i need to use?

I thought you wanted to do it from scratch? The standard libs will probably have all the math functions you need. + Something for keyboard and mouse input handling.

Where to start?

By reading something about 3d rendering. Triangle math. Projections. Lighting. Shaders. etc. You can look at university courses that publish their materials online. Or a book. Or blog guides. Or yt videos. Or stackoverflow. Or reddit posts. (quality drops aproximately in the order i've written it)

For a second I read this as "create my own 3D engine in Scratch" and was immediately both amused and horrified. 😆😵‍💫

Whatever language you want, and then math. Lots of matrix math. Get a book on the math behind 3D graphics.

Set yourself clear milestones. When will this project be "done"? What features is the engine going to have? What is in/out of scope?

And be aware, as an indie you either make a game engine or you make a game. It's near impossible to achieve both without sinking a crazy amount of time into it.

Making a game engine is an excellent way to sharpen your skillset when it comes to programming, math, physics, and graphics APIs. But it's an awful way to make a game. Just be sure you're going into it for the right reasons.

I was once a fool like you :)

Mike McShaffry’s book “Game Coding Complete” is a good guide to the practical side of using a game engine IRL to get things done.

It’ll give you a good idea of how things should be shaped in order to be useful, and some things you can “skip ahead” to. Off-the-shelf engines have to be extremely general in order to be flexible enough to be useful to many customers, so game devs have to put in the effort to make them more specific. You’ll have to start off by being specific, if you have any chance of actually finishing something.

Eberly’s book “3D Game Engine Architecture” deals with the nuts and bolts, the rigorous academic engineering stuff. It’s pretty solid, but it’s aimed at making a general-purpose engine, which is beyond the scope of a one-person project.

Backing up though… You don’t have any language or library opinions? You might need 5-10 years of experience doing general programming (or game dev) before you can sustainably tackle this, or else you’re likely to paint yourself into a corner.

Edit: Probably the biggest PITA with game engine dev is testing. If you’re not already an expert in setting up test harnesses at multiple levels of detail, you’re gonna find it impossible to keep moving after a few months.

Good luck!

https://scratchapixel.com/ for valuable background

After that, I recommend you learn webGL and write your first project in javascript. It's not as cool as a native executable in C++, but you will gain a lot of crucial 3D graphics knowledge without getting sidetracked by typical native development woes (which should not be underestimated)

Once you feel you have a good grasp on the 3D math, working with a graphics API, writing shaders, and the general architecture of a 3D application, THEN you should start looking into native development. Maybe start learning Vulkan as a next step, although that's a tough one if you don't already have C or C++ experience (mostly because of memory management concepts). Instead of Vulkan, you could instead go with OpenGL or a high level library like Sokol.

It should go without saying that this isn't an easy topic. It's a deep, deep (but rewarding) rabbit hole. There's a lot of stuff to learn, and it will take a while. A sense of overwhelming dread, hopelessness, and inadequacy is part of the process.

There are a couple of approaches you can take, depending on your learning style.

The first is that you could start really easy by creating a 1d engine, and then once you've gotten that mastered keep adding more dimensions until you get to 3d.

The second is that you could skip all the way to creating a 4d engine, and once you've gotten that mastered just chip off one dimension and bam you now have a 3d engine.

Start small first. Try to build a 2d engine

If you are using C, you could consider SDL 3 as a base library.

Not a recommendation though. Just saying you can do so.
It has all the basic stuff: V-Sync, audio, input device handling etc.

Almost forgot to add... It also allows you to easily add stuff with Vulkan, OpenGL etc.

If you already know some programming languages, look for some kind of GUI or game library for it to see if you can use it. If not, something like Blender might be easiest to make in C++, Rust, C (if you're a masochist), or maybe Zig. This may also influence the shading language you choose. Start with this.

You will need to know some shader language. You have a few options there, but the most popular are GLSL and OpenGL (though I'd prefer GLSL). There's also WGSL and some others, but they aren't as popular. Prefer whatever the graphics library you're using wants you to use.

Math is very heavy on linear algebra. Look up PBR if you want to render realistic 3d shapes. Google's Filament is well documented and walks through implementing it yourself if you want, but it's pretty advanced, so you might want to start simpler (fragment colors can just be base color * light color * light attenuation * (N*L) for example).

Use opengl and c or c++

I think NeHe might still have tutorials on this, in C/C++. You probably want to be using OpenGL for acceleration and maybe the old fashioned immediate mode/fixed function stuff where you call functions like "We are drawing triangles now" and "here is a vertex" and "that vertex is blue", and you can put off taking over the pipeline with your own shader code until later.

You still might be letting the library/gpu do most of "the maths" because I think you mostly hand it transformation matrices and points and it sends them to screen space. If you take over the vertex shader then your shader code does that.

You want to write a vector and matrix math library with 3-vectors and 4-vectors and 3x3 and 4x4 matrices, and add and multiply operations, and matrix inversion. The 4th dimension lets you make translation in 3D a linear multiply operation because you keep 1 in there and your matrix to represent a translation mixes that 1 into the other position coordinates to translate.

You also probably want to learn linear algebra enough for that to make sense.

And then on top of that you want to build a scene graph library where objects have parents they move with. And then your renderer loop will walk the scene graph node tree and push each object's transformation matrix and draw it and do its children and then pop the transformation matrix off again.