I'll make it perfectly clear that I'm not in anyway informed on the specific mechanisms or the mathematics involved with the algos related to LLMs. I have a computer science bachelor's and worked as a full stack developer. When I had a more free-form web-programming project due it was still called NLP and was used to assess webcrawler data and make recommendations: that's my entire hands on experience so that's why I'm sticking entirely to philosophical aspect.

Is it embodiment, is it the simplicity of artificial neurons compared to biological ones

That's basically it. I believe that by bowing to our machine god we aren't giving human intellect enough credit because we don't know and might never know how to measure it precisely enough to torture a machine with a soul. When you write an code the machine is always doing exactly what you told it to, it can't make mistakes. If your code fucked up, you fucked up. Therefore, without a total understand of our body I do not believe we'll be able to make a machine in our image because we don't have the full picture (please forgive me for that).

Even if you were to 1:1 model a neuron in code, you've got about the entire rest of the body left to recreate. Every thought and feeling is tied to your mind and body. It's not really "mind and body" it's really just "body". there are indescribable feelings that are expressed through song, poetry, paintings, dance, etc. There are entirely "illogical" processes that we simply cannot model. If we are ever able to, of course we can make a machine intelligent. If somehow we were able to get there I think the most obvious question is then: why? Someone on this site might be able to come up with a better use of our top minds and resources.