I'm not talking about "machines" or any other generic term.

I'm talking specifically about LLMs. And their limitations are evident. For instance, maths is one of the many things they can't do (and will never be able to do in any efficient way).

We have indeed, developed programs that play chess better than people (though sadly, until the LLM bubble pops we probably won't get any further). But they're not LLMs, or anything resembling an LLM. Because one of the other many things an LLM can't do is play games of skill. Or reason. Or solve puzzles. Or even have a concept of strategy.

LLMs, again, can only do one single thing. And that's to pick up the one card from their deck that's been picked up most often after the sequence of cards on the table according to their training model.

That's all they do. That's all they'll ever be able to do. Because that's how they work. And, sure, with that you can make it look like they're holding a conversation (until you ask them something that isn't in their model), but that's it.

They'll put words after another according to statistics (not, keep that in mind, according to meaning, or strategy, or anything like that; they don't, and can't know or care what the words mean, or whether the sentence they've put together makes any sense, or whether what it's stating is true or false), and that's that.

They won't play chess, they won't write good innovative code, they won't write original stories, and they won't drive your car.

We don't need to know how what we call consciousness works to know that. We just need to know how LLMs work. And that we most definitely do.