People being convinced that something is conscious is a long, long way from a compelling argument that something is conscious. People naturally anthropomorphize, and a reasonably accurate human speech predictor is a prime example of something that can be very easily anthropomorphized. It is also unsurprising that LLMs have developed such conceptual nodes; these concepts are fundamental to the human experience, thus undergird most human speech, and it is therefore not only unsurprising but expected that a system built to detect statistical patterns in human speech would identify these foundational concepts.

“So rocks are conscious” isn’t, at least in my opinion, the classic counter to panpsychism; it’s an attempt at reductio ad absurdum, but not a very good one, as the panpsychist can very easily fall back on the credible argument that consciousness comes in degrees, perhaps informed by systematic complexity, and so the consciousness of a rock is to the consciousness of a person as the mass of an atom is to the mass of a brain.

The problem with panpsychism is, and has always been, that there’s absolutely no reason to think that it’s true. It’s a pleasingly neat solution to Chalmers’ “hard problem” of neuroscience, but ultimately just as baseless as positing the existence of an all-powerful God through whose grace we are granted consciousness; that is, it rests on a premise that, while sufficiently explanatory, is neither provable nor disprovable.

We ultimately have absolutely no idea how consciousness arises from physical matter. It is possible that we cannot know, and that the mechanism is hidden in facets of reality that the human experience is not equipped to parse. It is also possible that, given sufficiently advanced neuroscience, we will be able to offer a compelling account of how human consciousness arises. Then—and only then—will we be in a position to credibly offer arguments about machine intelligence. Until then, it is simply a matter of faith. The believers will see a sufficiently advanced language model and convince themselves that there is no way such a thing is not conscious, and the disbelievers will repeat the same tired arguments resting on the notion that a lack of proof is tantamount to a disproof.