[Double reply to avoid editing my earlier comment]

From the HN thread:

It's a good example how this models are not answering based on any form of understanding and logic reasoning but probabilistic likelihood in many overlapping layers. // Through this also may not matter if this creates a good enough illusion of understanding and intelligence.

I think that the first sentence is accurate, but I disagree with the second one.

Probabilistic likelihood is not enough to create a good illusion of understanding/intelligence. Relying on it will create situations as in the OP, where the bot outputs nonsense because of an unexpected prompt.

To avoid that, the model would need some symbolic (or semantic, or conceptual) layer[s], and handle the concepts being conveyed by the tokens, not just the tokens themselves. But that's already closer to intelligence than to prob likelihood.