It does not create "thoughts", it is very good at tricking humans into believing that it does, though.

You talk about how human cognition is more complex and squishy, but nobody really knows how it truly works inside.

It is not that there is no understanding, but rather that we have incomplete understanding. We know, for example, that human cognition is not purely storing recorded stimuli and performing associative analysis against them when meeting other stimuli.

All I see is the same kind of blackbox. A kid trying many, many times to stand up, or to say "papa", until it somehow works, and now the pathway is setup in the brain.

This is a bit of a logical fallacy here, unfortunately, specifically false equivalency (ie. Thing A and Thing B both have characteristic C, therefore Thing A and Thing B are the same). This is exactly the sort of "dangerous" fallacy that a number of AI academics have warned about as well. LLMs are great at producing outputs that our socially-oriented brains can interpret as sentient thought and mistakenly anthropomorphize.

However, LLMs, as the word "model" in the name suggests, are statistical modeling software. They do not understand context or abstract meaning; only statistical occurrence of data in their stack, compared to the inputs. They are physically incapable of developing the Theory of the Mind due to the limitations in how they work.

But does it make it NOT intelligent?

No. The fact that they literally cannot actually understand anything or undertake contemplative, abstract thoughts is what makes them not intelligent. They do not understand the meaning of language; it is just data to them that has no context but how it relates to other parts of language.

Yeah, maybe it does not work like our brain.

I absolutely think that LLMs could be a component in AI but, alone, they are just like saying that a tire is a car because both can travel linear distances using rotation movements. By themselves, LLMs fail to fulfill what we tend to define as intelligence.

But is a human brain structure the only possible structure for intelligence? I don't think so.

I certainly hope that the human brain isn't the only possible structure for intelligence and find it very unlikely because our meat-computers aren't really that special, even if we can't entirely understand how they work yet (we've only really been trying for a relatively short time, compared to our species' existence). We seem to agree there. I absolutely want AI as well as other non-human intelligence to be a thing because the idea of a universe in which humanity is the only sentience is very lonely and sad to me.

If you consider the amount of text an LLM has to consume to replicate something approaching human like language you have to appreciate there is something else going on with our cognition. LLM's give responses that make statistical sense but humans can actually understand why one arrangement of words might not make sense over the other.