The Chinese Room is really a thought experiment about the inner workings of a partner in a Turing test. Externally they have the same pitfalls, but the Chinese Room also reveals itself completely if one can observe in detail the inner workings of the room/partner.

LLMs are still mostly black boxes, but we can have enough of a glimpse inside to reveal that they aren't "following some rails" like a simple algorithm.

make mistakes such as accidentally copying out the response next to the correct response and still make sense

Precisely. This is another part that we can see with LLMs: at runtime, the models get applied a "temperature" parameter, which intentionally introduces a certain level of mistakes. With "temperature = 0", the output is a "stochastic parrot", and quickly turns into nonsense. With a higher temperature, the randomness increases and the output becomes a total mess. But setting it just right, to a sweet spot of "very little, but not zero", turns out to produce the outputs that we see in ChatGPT and similar.

Knowing that the concept space of LLMs has similar concepts clustered, it makes sense that these errors would force the LLM to sometimes make associations on the fly between close concepts, associations that it didn't have trained for before, and which "derail" it into a close, but not exactly the same, train of thought.

This behavior also seems to be what we call "intelligence" in humans: the ability to solve problems not seen before (zero shot).

A further extension would be the ability to constantly learn from every interaction. Right now LLMs have a "context" of some length, that changes dynamically, but has no influence over the pre-trained network.

Interestingly, this has a parallel in "crystallized intelligence" vs. "fluid intelligence" in humans.

So... maybe LLMs are not full AGIs yet, but they are showing many of the behaviors that we would expect from an AGI, while at the same time giving or confirming insights into the workings of the human mind itself.