The LLM isn’t trained to be reliable, it’s trained to be confident.

And it's promoted by business people with the exact same skill set who have been rewarded for it. I would argue though that there's nothing wrong with what LLMs are doing: they're doing what they were trained to do. The con is in how the confidently unreliable techbros sell it to us as a source of knowledge and understanding akin to a search engine, when it's nothing of the sort.

an llm is a cool way to rephrase your own thoughts back at you. it's pretty useful for brainstorming. or masturbation. i sure hope that's all anyone uses it for

Confidence mixed with a lack of domain knowledge is a tale as old as time. There's not always a con in play -- think Pizzagate -- but this certainly isn't restricted to LLMs, and given the training corpus, a lot of that shit is going to slip in.

It's really unclear where we go from here, other than it won't be good.

AI can be useful without being right about everything. But the user has to know enough to push back or just write it themselves when necessary. And in my experience the same is true when pairing with another developer, too.

It's a tool, not a solution. Though it's valid to say the folks touting its miracle capabilities are full of shit. It is imperfect, but it's not worthless. It's not a con man, it's just confidently wrong. I've worked with/for a lot of people like that.