If I say, "Knight to B4," does that sound like something a person playing chess might say? Then it did it's job.

Think of an LLM as an actor. You don't hire someone to act as a grandmaster in a movie based on their skill at chess, they might not even know how to play, but if they deliver the lines in a convincing way, that's what you're looking for. There's chess AIs that are incredibly good at chess, because that's what they're designed for and trained on. That's why this is a very silly test, it's like testing a fish on its tree-climbing ability, the only thing sillier than this test is that people are surprised by it.