I am deeply troubled by the way that AIs slip right past peoples' defenses and convince them of things that are absolutely not true. Not just things like the AI psychosis that some people have been driven into, not just the hallucinations or overly fawning over terrible ideas, it goes so much further than our monkey brains can understand. These things do not think, they do not have feelings, they don't have motivations, they don't have morals or values, no sense of right or wrong, they are, quite literally, word prediction machines that are selecting their responses semi-randomly. And yet, even people who know this to be the case cannot stop themselves from anthropomorphizing the AI. All of our human instincts scream "this is a person!" when we interact with them, and with that comes an assumption of values, morals, thoughts, and feelings, none of which are there. There is a fundamental mismatch between the human mental model and the reality of the software, and that is harmful, even dangerous. We will give benefit of the doubt to them, we will believe their explanations about "why" they "chose" to say or do what they did, and we will repeatedly make the same fundamental mistakes because we want to believe a total lie.

And that's not even getting in to the harm when they are working properly, encouraging us to outsource our thinking and creativity to a service that charges monthly. I'm seriously worried that kids in school now are going to have their learning stunted terribly, it's just so easy to have any and all homework done in a matter of minutes without learning a single thing.