This is exactly correct, except you’re also not accounting for the insane amount of computational power that would be necessary to backtrack a single output of a single model. This is why it is a black box. It simply is not possible on a meaningful level.

It's not practical, but it can be done. We simple don't have the time or inclination to do it.

It's like like saying we don't understand how an internal combustion engine works. Every explosion is a bit different, it pushes the pistons a bit less or more, it leaves a bit more or less residue in different places. We can't backtrack and check every cycle and every part on a meaningful level, but we understand of it works, and we could do it if we wanted to. It's just not practical.

As an example from my field: if you damage the dorsolateral prefrontal cortex in a fully grown adult, they will have the impulse control of a three-year old. We know this because we have observed damage to this area in multiple individuals, and can measure the effects based on the severity of that damage.

Okay so explain how sudden savant syndrome works. Step by step, biochemical process by biochemical process.

In contrast, if you provide the same billion-parameter neural network identical inputs, you will not receive identical outputs.

If you take the same model, put it in a VM, give it an input, get an output and the restore the VM to the exact same state before and ensure there's no randomness, the model will give you same output.