It's because the LLM is incapable of understanding symbols, so it couldn't even understand the chessboard and the images that represented the pieces. This capability for abstract thinking is the thing that human brains do incredibly well (sometimes too well, then you get pareidolia), but is completely outside the bounds of what an LLM is or ever will be able to do.

I'm sure they've digested every public piece of chess notation ever written but they have no capacity for comprehension and are programs that emit text shaped like chess notation if you make that request of them.

when people here call it a text extrusion machine thats literally what it is. In fact it doesnt even look at text, it looks at tokens. And there are a limited number of tokens (llama uses a vocabulary size of about 32k i think). It takes all of the previously entered input and output, turns it into tokens, and then each token “attends” (is multiplied by with some coefficient) to each other token. Then it all goes through more gigantic layers of matrix multiplication and at the end you have the statistically most likely next token. Then it does the whole thing again recursively until it reaches what it decides is the end of the output. It may also not decide and would need to be cut off.

So its not really looking at the game. It is in a way but it doesnt really know the rules, its just producing the next most likely token which is not necrssarily the next best move or even next correct move.

An LLM can summarize the rules of chess, because it predicts the sequence of words needed to create that with incredible accuracy. This is why it’s so weird when it goes wrong, because if one part of it is off then it throws the rest of the work it’s doing out of balance.

But all it is doing is a statistical analysis of all the writing it’s has been trained on and determining the best next word to use (some later models do them in groups and out of order).

That doesn’t tell it fuck-all about how to make a chess move. It’s not ingesting information in a way that lets it create a model to tell you what the next best chess move is, how to solve linear algebra, or any other activity that requires procedural thought.

It’s just a chatterbox that tells you whatever you want to hear. No wonder the chuds love it

If I say, "Knight to B4," does that sound like something a person playing chess might say? Then it did it's job.

Think of an LLM as an actor. You don't hire someone to act as a grandmaster in a movie based on their skill at chess, they might not even know how to play, but if they deliver the lines in a convincing way, that's what you're looking for. There's chess AIs that are incredibly good at chess, because that's what they're designed for and trained on. That's why this is a very silly test, it's like testing a fish on its tree-climbing ability, the only thing sillier than this test is that people are surprised by it.