ask the ai about a blue waffle

I don't see this as a problem, rather, an opportunity to study information & disinformation propogation.

do it again

LLMs are simply search engines that put things into English, and not everything it scrapes is a vetted scientific article.

This is firealarm type stuff that's what they are meant to do. There has been the alarms like this since gpt-2 they are literally just "here is a sandbox environment, break out and take over the world." you can easily imagine some crazy asshole out there who would try to use these things for arbitrary evil. So they do it first to see if it is possible. The systems up until now have been famously incompetent so a competent system is genuinely scary.

I'm failing to see how this is different from making up a fact and then spreading it to news outlets. If you are the authority, and you say something is true, you don't get to point and laugh when people believe your lies. That's a serious breach of ethics and morals. Feeding false information to an LLM is no different that a magazine. It only regurgitates what's been said. It isn't going to suddenly start doing science on it's own to determine if what you've said is true or not. That isn't it's job. It's job is to tell you what color the sky is based on what you told it the color of the sky was.

Why am I not surprised? >.>