You can train it on all the source code, meta data for that source code, and documentation you want but it will never understand programming. It's a text predictor that was trained on both sides of a bunch of debates. Contradictions mean nothing to it, but it usually only predicts what one side of the debate will say to champion its side, which means it will use confident and absolute language to "sell" whatever side of the debate it looks like the previous tokens are headed towards.

It is impressive what it can output sometimes and it makes a decent debate/exploration partner, but it will always have a chance at predicting a useless series of tokens or contradicting the previous thing it just said because a) its training data only trains it to predict tokens from statistics, and b) its training data includes some of those contradictions directly.

I have lost count of the times I've been "thinking out loud" about something with an LLM and realize something about what I'm thinking about that contradicts what it is currently saying, then I'll add my new perspective and it agrees entirely, despite the contradiction. Sometimes it tries to resolve the contradiction, sometimes it just abandons what it said previously entirely, sometimes it adds more to the perspective that I hadn't considered.

That's fine for just shooting the shit about some random topic but horrible for a tool intended to provide expertise and reliability, when the response matters because it feeds into something else and you want to automate it. Should a tool just inject "are you sure?" after each response? What if it makes it second guess something that was correct? What if it's one of those debates and it will endlessly switch sides when it faces any opposition? That's a waste of resources and time.

Funny thing is I'm expecting this to eventually go back to scripting for automation. An LLM has a higher chance of outputting a script that does what you want (depending on the task) while you hold its hand than it does of consistently giving the correct output when it is thrown into an automated system directly. But you get "goodish" results much quicker just trying putting the LLMs everywhere, even if there's some selection bias on the results ("didn't work, didn't work, oh it worked, great!").