That's a common misunderstanding.

LLMs have billions of neurons, and we can see firsthand how information travels along their neural pathways and, yeah, it looks a whole lot like they're thinking. If anything, we have more concrete proof that LLMs think than that humans think.

They do think, it's just that they don't have short term memory. They can only remember things linguistically, by talking and then listening to their own output. It's an artifact of how we've set them up to interact with the world. Many humans use a similar thought process for certain problems (e.g. talking out loud to a rubber duck). Sure, there are other ways humans think too (e.g. visual/spatial), but linguistic thought is still valid.