Of course, the training data contains all that information, and the LLM is able to explain it in a thousand different ways until anyone can understand it.

But flip that around.

You could never explain a brand new concept to an LLM which isn't already contained somewhere in its training data. You can't just give it a book about a new thing, or have a conversation about it, and then have it understand it.

A single book isn't enough. It needs terabytes of redundant examples and centuries of cpu-time to model the relevant concepts.

Where a human can read a single physics book, and then write part 2 that re-explains and perhaps explores new extrapolated phenomenon, an LLM cannot.

Write a completely new OS that works in a completely new way, and there is no way you could ever get an LLM to understand it by just talking to it. To train it, you'd need to produce those several terabytes of training data about it, first.

And once you do, how do you know it isn't just pseudo-plagiarizing the contents of that training data?