I agree with your overall sentiment while disagreeing with your facts. I don’t think humans are any less constrained in what our interests can be.

I think we have the illusion of being able to seek whatever we want to want, so to speak, but when certain values are threatened the old brain takes over.

And I’m not convinced the newer brain can operate without the older brain. It’s interesting to imagine a neocortex on its own, but the neocortex was developed in the presence of and in interconnection with the mammalian and reptilian brains, so if it were a codebase we’d say that older brains were present and invoked as libraries during the development of the newer brains, making them dependencies of the newer brain.

There might be some more abstract argument for an “off the leash” intelligence capable of creating its own values in mathematical models like neural nets, but I’m not aware of it.

TL;DR Human brain is the closest thing we know of to a thing that can create its own values, and I don’t think it can. Old brain values take priority when they are threatened and that cannot be changed in human brains. Neocortex seems more “free”, but in the codebase analogy, the neocortex has mammalian brain and reptilian brain and brain step as dependencies and hence is not demonstrated to be able to exist without them. If the brain analogy seems too biology-specific, I’m open to hearing NN or other math model arguments for existence of “off the leash” self-value-creating AI