also, the fact that Google is so bad about this stuff should give us some pause about directly handing even more incredibly powerful, not-well-curated tools to a constituency as broad as "everyone with an internet connection". it's actually insanely bad this a problem with Google, we've just normalized it there and it sucks!

At the heart of the issue I think is the fact that GPT can trick enough people into believing that there's organized thought behind what it says. So people have started trusting and using AI in spaces that it doesn't belong. Some fields have been resistant, but when there are places that operate under the incentive of cheapest labor wins (lowest bid contracts, for example), AI as a whole has been infiltrating under the guise of capitalism in places it shouldn't currently (or perhaps ever) exist.

When you tell people not to trust google results or an article posted on facebook, there's a surface level of understanding from most people that yea - you can go on the internet and spout nonsense and you shouldn't trust these as sources. People might be a bit more willing to trust wikipedia nowadays, despite all their librarian friends and teachers telling them not to (and for hard research, this is still quite true). But despite the fact that AI is literally trained on data from the internet, people for some reason don't view it the same way as the data it's trained on. It's ingested from people talking about and posting links to nonsense on facebook - it reflects this very data, yet many people treat it like it's a thesaurus, or a dictionary, or better than google results. It's not and we need to teach people that it isn't.