Just want to point out that you shouldn't use them as a single point of truth. They can and do give factual information, but just as with every source, you should verify and know what you're actually doing with the output. Part of why it's so useful to me as a programmer is that I can determine between good and bad outputs and utilize the good ones.

the argument they’re making is similar to Godel’s Incompleteness Theorem

More like the conclusion. I will be shocked if I read this and they're doing Peano arithmetic.

Indeed. I frequently use LLMs as brainstorming buddies while working on creative things, like RPG adventure planning and character creation. I want the AI to come up with new and unexpected things that never existed before.

If I have need of the AI to account for "ground truths" then I use things like retrieval-augmented generation or database plugins that inject that stuff into the context.

I mean, naively you'd expect it to always hallucinate/lie, but it does give real facts at least some of the time.

The issue is not that it doesn’t know everything, it’s that it doesn’t know anything. It’s not capable of knowledge in the sense that humans are. All it does is probabilistically predict which sequence of words might best respond to a prompt, based on huge amounts of human text that it was trained on.

Part of the issue is how will you train the model to know which things in its training data are factual and which are not? An incredible amount of human curation already goes into just avoiding the model from repeating offensive things, but the realm of facts is so so much broader than that. I don’t see any way it could be done.

But on the other hand I am only a casual observer of this technology and perhaps the experts will come up with a creative solution we can’t yet imagine.

That's not how LLMs work.

Super short version is that LLMs probabilistically determine the next word most likely to occur in a sequence. They do this using Statistical Models (like what your cell phone's auto complete uses); Transformers (rating the importance of preceding words, so the model can "focus" on the most important words); and Relatedness (a measure of how closely linked different words/phrases are to reach other in meaning).

With increasingly large models, LLMs can build a more accurate representation of Relatedness across a wider range of topics. With enough examples, LLMs can infinitely generate content that is closely Related to a query.

So a small LLM can make sentences that follow writing conventions but are nonsense. A larger LLM can write intelligibly about topics that are frequently included in the training materials. Huge LLMs can do increasingly nuanced things like "explain" jokes.

LLMs are not capable of evaluating truth or facts. It's not part of the algorithm. And it doesn't matter how big they get. At best, with enough examples to build a stronger Relatedness dataset, they are more likely to "stay on topic" and return results that are actually similar to what is being asked.

two totally independent LLMs

How do you propose to get these independent LLMs? If both are trained using similar objectives e.g., masked token prediction, then they won’t be independent.

Also, assuming independent LLMs could be obtained, how do you propose to compute this hallucination probability? Without knowing this probability, you can’t know how many verification LLMs are sufficient for your application, can you?