LLMs have been around since roughly ~~2016~~ 2017 (comment below corrected me that Attention paper was 2017). While scaling the up has improved their performance/capabilities, there are fundamental limitations on the actual approach. Behind the scenes, LLMs (even multimodal ones like gpt4) are trying to predict what is most expected, while that can be powerful it means they can never innovate or be truth systems.

For years we used things like tf-idf to vectorize words, then embeddings, now transformers (supped up embeddings). Each approach has it limits, LLMs are no different. The results we see now are surprisingly good, but don't overcome the baseline limitations in the underlying model.

I'm not trained in formal computer science, so I'm unable to evaluate the quality of this paper's argument, but there's a preprint out that claims to prove that current computing architectures will never be able to advance to AGI, and that rather than accelerating, improvements are only going to slow down due to the exponential increase in resources necessary for any incremental advancements (because it's an NP-hard problem). That doesn't prove LLMs are end of the line, but it does suggest that additional improvements are likely to be marginal.

Reclaiming AI as a theoretical tool for cognitive science

we're extremely early on

Oh really! The analysis has been established since the 80's. Its so far from early on that statement is comical