Except we already have fields (like pharma manufacturing) that have to deal with hundreds or thousands of inputs and variables, are automated, and we still manage to fully understand the stack as well as fully check everything.

Hint: when someone tells you they "can't" check or understand what their software is doing, it's a scam.

Normally they should be told to go back and figure it out before being allowed to ship any product. If you tried this in any other industry it would be laughable. Even in software it's outrageous, imagine getting accounting software or even a simple file backup tool that doesn't work some of the time and nobody can tell you how it works. Yet these companies get a pass putting cars like this on the road.