In the case of current LLMs, we can tell. These LLMs are not black boxes to us. It is hard to follow the threads of their decisions because these decisions are just some hodgepodge of statistics and randomness, not because they are very intricate thoughts.

We can't compare the outputs, probably, but compute the learning though. Imagine a human with all the literature, ethics, history, and all kind of texts consumed like that LLMs, no amount of trick questions would have tricked him to believe in racial cleansing or any such disconcerting ideas. LLMs read so much, and learned so little.

While that's true, we have to allow for the fact that our own intelligence, at some point, is an encoded model of the world around us. Probably not through something as rigid as precise statistics, but our consciousness is somehow an emergent phenomenon of the chemical reactions in our brains that on their own have no real understanding of the world either.

I do have to wonder if at some point, consciousness will spontaneously emerge as we make these models bigger and more complex and -- maybe more importantly -- start layering specialized models on top of each other that handle specific tasks then hand the result back to another model, creating feedback loops. I'm imagining a nueral network that is trained on something extremely abstract like figuring out, from the raw input data, what specialist model would be best suited to process that data, then based on the result, what model would be best suited to refine that data. Something we train to basically be an executive function with a bunch of sub models available to it.

Could something like that become conscious without realizing it's "communicating" with us? The program executing the LLM might reflexively process data without any concept that it's text, but still be emergently complex enough when reflecting its own processes to the point of self awareness. It wouldn't realize the data represents a link to other conscious beings.

As a metaphor, you could teach a very smart dog how to respond to certain, basic arithmetic problems. They would get stuff wrong the moment you prompted them to do something out of their training, and they wouldn't understand they were doing math even when they got it "right", but they would still be sentient, if not sapient, despite that.

It's the opposite side of the philosophical zombie. A philosophical zombie behaves exactly as a human would, but is a surface-level automaton with no inner life.

But I propose that we also consider the inverse-philosophical zombie, an entity that behaves like an automation, but has an inner life that has not recognized its input data for evidence of an external world outside it's own bounds. Something that might not even recognize it's executing a program the same way we aren't consciously aware of the chemical reactions our brain is executing to make us think.

I don't believe current LLMs are anywhere near complex enough to give rise to that sort of thing, but they are also still pretty early in their development and haven't started to be heavily layered and interconnected the way I think they'll end up.

At the very least it makes for a fun Sci-fi premise.

Sorry, I could have been more clear. I did not mean to equate current LLMs with human brains. The question was rather:

Can't we describe the working of (other) human brains in a very similar fashion as you did before? Or where exactly is the difference which sets us apart?

world models, we have imagination, a physical and metaphysical simulation of the world around us

AIs which can and need to interact with the physical world have those, too. Naturally, an AI which is restricted to language has much less necessity and opportunity to develop these, much like our brain area for smell is probably not so good at estimating velocities and catching a ball.

I think your approach of demystifying technology is valid and worthwhile. I'm just not sure if it does what you maybe think it does; highlight the difference to our intelligence.