This is a perfect demonstration of how LLMs work and why they do not think.

The base question here, that the model is most strongly statistically geared towards, is "How many Rs are in strawberry". You can see how the response in the screenshot works as the template for the correct answer to this question.

All it did was get the most likely response for the strawberry question (which is the closest, most confident match in structure to the blueberry question) , and then substitute specific tokens. This is essentially what it does with every response for any question. It uses the closest match from the data it is trained on, then substitutes individual terms, so it looks appropriate to the question.

Ultimately every answer will only ever be an approximation, but there will never be any certainty to its correctness.

For reference, the reason why this happens is because LLMs aren't "next word predictors", but rather "next token predictors". Each word is broken into tokens, probably 'blue' and 'berry' for this case. The LLM doesn't have any access to information below the token level, which means that it can't count letters directly, but it has to rely on the "proximity" of the tokens in it's training data. Because there's a lot on the Internet about letters and strawberries, it counts the r instead of the b in 'berry'. Chain of Thought (CoT) models like Deepseek-reasoner or ChatGPT-o3 feed their output back into themselves and are more likely to output the text 'b l u e b e r r y' which is the trick to doing this. The lack of sub-token information isn't a critical flaw and doesn't come up often in real world usecases, so there isn't much energy dedicated to fixing it.

The entire economy is getting refocused onto building a robot that lies to you.

OpenAI got that sweet DoD gig and now they're just slapping a UI wrapper on GPT 3.5 and calling it GPT 5.

Well yes it's terrible and hallucinates, it's a real piece of shit actually, but you see of course this is precisely why we need to commit all of humanity's resources. To improve it! To allow it to spell a word!

I wonder if I would convince dumb rich investors to buy bags of my poop as the next big innovation

AI has its utilities, but capitalists searching for new frontiers and trying to find a genie that can solve climate change, poverty, wealth inequality, and really all of humanity's problems - directly and indirectly caused by capitalism - is not going to happen.

every time i see the failures of the fancy predictive text machine i find myself asking "what exactly was wrong with expert systems?"
like, they actually work for what people need them for?

When is the screenshot from and which model?

I wanna see the results if you ask ChatGPT the same question a million times. What percentage of responses would actually get the correct number?

AI general intelligence acheived, I've probably answered this same question with that answer at some point in my life. And I have some level of intelligence.

3 Rs in strawberry

Strawberry and blueberry are both in "berry" category and are more closely associated with each other than any other fruit

B is to Blueberry the way R is to Strawberry

Therefore, blueberry has 3 Bs