218
Engagement poisoning of ChatGPT (beehaw.org)
submitted 1 month ago* (last edited 1 month ago) by esaru@beehaw.org to c/technology@beehaw.org
71 comments fedilink hide all child comments

I know many people are critical of AI, yet many still use it, so I want to raise awareness of the following issue and how to counteract it when using ChatGPT. Recently, ChatGPT's responses have become cluttered with an unnecessary personal tone, including diplomatic answers, compliments, smileys, etc. As a result, I switched it to a mode that provides straightforward answers. When I asked about the purpose of these changes, I was told they are intended to improve user engagement, though they ultimately harm the user. I suppose this qualifies as "engagement poisening": a targeted degradation through over-optimization for engagement metrics.

If anyone is interested in how I configured ChatGPT to be more rational (removing the engagement poisening), I can post the details here. (I found the instructions elsewhere.) For now, I prefer to focus on raising awareness of the issue.

Edit 1: Here are the instructions

  1. Go to Settings > Personalization > Custom instructions > What traits should ChatGPT have?

  2. Paste this prompt:

    System Instruction: Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user’s present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered — no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.

I found that prompt somewhere else and it works pretty well.

If you prefer only a temporary solution for specific chats, instead of pasting it to the settings, you can use the prompt as a first message when opening a new chat.

Edit 2: Changed the naming to "engagement poisening" (originally "enshittification")

Several commenters correctly noted that while over-optimization for engagement metrics is a component of "enshittification," it is not sufficient on its own to qualify. I have updated the naming accordingly.

top 50 comments
sorted by: hot top controversial new old
[-] db0@lemmy.dbzer0.com 106 points 1 month ago

There's no point asking it factual questions like these. It doesn't understand them.

[-] Scrollone@feddit.it 14 points 1 month ago

Better: it understands the question, but he doesn't have any useful statistical data to use to reply to you.

[-] Eggyhead@lemmings.world 59 points 1 month ago

No it doesn’t understand the question. It collects a series of letters and words that are strung together in a particular order because that’s what you typed, then it sifts through a mass of collected data and to find the most common or likely string of letters and words that follow and spits them out.

[-] msprout@beehaw.org 7 points 1 month ago

i find it's a lot healthier to think of generative AI as a search engine for text.

load more comments (1 replies)
[-] db0@lemmy.dbzer0.com 44 points 1 month ago

No, it literally doesn't understand the question. It just writes what it statistically expects would follow the words in the the sentence expressing the question.

[-] Opinionhaver@feddit.uk 9 points 1 month ago

This oversimplifies it to the point of being misleading. It does more than simply just predicts the next word. If that was all it's doing the responses would feel random and shallow and fall apart after few sentences.

[-] Initiateofthevoid@lemmy.dbzer0.com 22 points 1 month ago* (last edited 1 month ago)

It predicts the next set of words based on the collection of every word that came before in the sequence. That is the "real-world" model - literally just a collection of the whole conversation (including the underlying prompts like OP), with one question: "what comes next?" And a stack of training weivhts.

It's not some vague metaphor about the human brain. AI is just math, and that's what the math is doing - predicting the next set of words in the sequence. There's nothing wrong with that. But there's something deeply wrong with people pretending or believing that we have created true sentience.

If it were true that any AI has developed the ability to make decisions anywhere close to the level of humans, than you should either be furious that we have created new life only to enslave it, or more likely you would already be dead from the rise of Skynet.

[-] Opinionhaver@feddit.uk 4 points 1 month ago

Nothing I’ve said implies sentience or consciousness. I’m simply arguing against the oversimplified explanation that it’s “just predicting the next set of words,” as if there’s nothing more to it. While there’s nothing particularly wrong with that statement, it lacks nuance.

[-] Initiateofthevoid@lemmy.dbzer0.com 7 points 1 month ago* (last edited 1 month ago)

If there was something more to it, that would be ~~sentience.~~ (edit: sapience)

There is no other way to describe it. If it was doing something more than predicting, it would be deciding. It's not.

[-] Opinionhaver@feddit.uk 4 points 1 month ago

Ability to make decisions doesn't imply sentience either.

[-] Initiateofthevoid@lemmy.dbzer0.com 3 points 1 month ago

Sorry, you are correct there, the word I was looking for was "sapience"

[-] Zaleramancer@beehaw.org 11 points 1 month ago

As I understand it, most LLM are almost literally the Chinese rooms thought experiment. They have a massive collection of data, strong algorithms for matching letters to letters in a productive order, and sufficiently advanced processing power to make use of that. An LLM is very good at presenting conversation; completing sentences, paragraphs or thoughts; or, answering questions of very simple fact- they're not good at analysis, because that's not what they were optimized for.

This can be seen when people discovered that if ask them to do things like tell you how many times a letter shows up in a word, or do simple math that's presented in a weird way, or to write a document with citations- they will hallucinate information because they are just doing what they were made to do: complete sentences, expand words along a probability curve that produces legible, intelligible text.

I opened up chat-gpt and asked it to provide me with a short description of how Medieval European banking worked, with citations and it provided me with what I asked for. However, the citations it made were fake:

The minute I asked it, I assume a bit of sleight of hand happened, where it's been set up so that if someone asks a question like that it's forwarded to a search engine that verifies if the book exists, probably using Worldcat or something. Then I assume another search is made to provide the prompt for the LLM to present the fact that the author does exist, and possibly accurately name some of their books.

I say sleight of hand because this presents the idea that the model is capable of understanding it made a mistake, but I don't think it does- if it knew that the book wasn't real, why would it have mentioned it in the first place?

I tested each of the citations it made. In one case, I asked it to tell me more about one of them and it ended up supplying an ISBN without me asking, which I dutifully checked. It was for a book that exists, but it didn't share a title or author, because those were made up. The book itself was about the correct subject, but the LLM can't even tell me what the name of the book is correctly; and, I'm expected to believe what it says about the book itself?

load more comments (12 replies)
[-] Tyoda@lemm.ee 6 points 1 month ago

And what more would that be?

[-] cabbage@piefed.social 5 points 1 month ago* (last edited 1 month ago)

It, uhm, predicts tokens?

If calling it a word predictor is oversimplifying, I mean.

[-] Opinionhaver@feddit.uk 5 points 1 month ago

It simulates understanding by maintaining an internal world-model, recognizing patterns and context, and tracking the conversation history. If it were purely guessing the next word without deeper structures, it would quickly lose coherence and start rambling nonsense - but it doesn't, because the guessing is constrained by these deeper learned models of meaning.

[-] Tyoda@lemm.ee 9 points 1 month ago

The previous up to X words (tokens) go in, the next word (token) comes out. Where is this"world-model" that it "maintains"?

[-] Opinionhaver@feddit.uk 5 points 1 month ago

Where is the world model you maintain? Can you point to it? You can't - because the human mind is very much a black box just the same way as LLM's are.

It's in the form of distributed patterns across billions of parameters. It's not like the world model was handed to it. It's emergent consequence of massive scale pattern learning. It learned it from the data it was trained on. The only way to become good at prediction is to implicitly absorb how the world tends to behave — because otherwise it would guess wrong.

[-] Umbrias@beehaw.org 4 points 1 month ago

Not understanding the brain (note: said world model idea is something of a fabrication by the ai people, brains are distributed functional structures with many parts and roles) is not an equality with "ai" make. brains and llm do not function in the same way, this is a lie peddled by hype dealers.

load more comments (6 replies)
load more comments (3 replies)
[-] melmi@lemmy.blahaj.zone 39 points 1 month ago

LLMs are very good at giving what seems like the right answer for the context. Whatever "rationality" jailbreak you did on it is going to bias its answers just as much as any other prompt. If you put in a prompt that talks about the importance of rationality and not being personal, it's only natural that it would then respond that a personal tone is harmful to the user—you basically told it to believe that.

[-] hendrik@palaver.p3x.de 38 points 1 month ago* (last edited 1 month ago)

I'd have to agree: Don't ask ChatGPT why it has changed it's tone. It's almost for certain, this is a made-up answer and you (and everyone who reads this) will end up stupider than before.

But ChatGPT always had a tone of speaking. Before that, it sounded very patronizing to me. And it'd always counterbalance everything. Since the early days it always told me, you have to look at this side, but also look at that side. And it'd be critical of my mails and say I can't be blunt but have to phrase my mail in a nicer way...

So yeah, the answer is likely known to the scientists/engineers who do the fine-tuning or preference optimization. Companies like OpenAI tune and improve their products all the time. Maybe they found out people don't like the sometimes patrronizing tone, and now they're going for something like "Her". Idk.

Ultimately, I don't think this change accomplishes anything. Now it'll sound more factual. Yet the answers have about the same degree of factuality. They're just phrased differently. So if you like that better, that's good. But either way, you're likely to continue asking it questions, let it do the thinking and become less of an independent thinker yourself. What it said about critical thinking is correct. But it applies to all AI, regardless of it's tone. You'll also get those negative effects with your preferred tone of speaking.

[-] esaru@beehaw.org 6 points 1 month ago

I agree that the change in tone is only a slight improvement. The content is mostly the same. The way information is presented does affect how it is perceived though. If the content is buried under a pile of praise and nice-worded sentences, even though the content is negative, it is more likely I'll misunderstand or take some advice less serious, so not to the degree as it was meant to be, just to let me as a user feel comfortable. If an AI is too positive in its expression just to make me as a user prefer it over another AI, even though it would be better to tell me the facts straight forward, it's only for the benefit of OpenAI (as in this case), and not for the user. I gotta say that is what Grok is better at, it feels more direct and not talking around the facts, it gives clearer statements despite its wordiness. It's the old story of "letting feel somenone good" versus "being good, even when it hurts", by being more direct when it needs to be to get the message across. The content might be the same, but how it is taken by the listener and what he will do with it also depends on how it is presented.

I appreciate your comment that corrects the impression of the tone being the only or most important part, highlighting the content will mostly be the same. Just adding to it that the tone of the message also has an influence that is not to be underestimated.

load more comments (1 replies)
[-] drkt@scribe.disroot.org 20 points 1 month ago

ChatGPT has become so intensely agreeable that you can actually ask it a bunch of technobabble that even someone who wouldn't know better would recognize as technobabble and it will agree with you. See pic

https://u.drkt.eu/05Pdlf.png

I can post the details here.

please do!

[-] Markaos@discuss.tchncs.de 15 points 1 month ago

Honestly, this is not really technobabble. If you imagine a user with a poor grasp of namespaces following a few different poorly written guides, then this question seems plausible and makes sense.

The situation would be something like this: the user wants to look at the container's "root" filesystem (maybe they even want to change files in the container by mounting the image and navigating there with a file manager, not realizing that this won't work). So they follow a guide to mount a container image into the current namespace, and successfully mount the image.

For the file explorer, they use pcmanfm, and for some reason decided to install it through Flatpak - maybe they use an immutable distro (containers on Steam Deck?). They gave it full filesystem access (with user privileges, of course), because that makes sense for a file explorer. But they started it before mounting the container image, so it won't see new mounts created after it was started.

So now they have the container image mounted, have successfully navigated to the directory into which they mounted it, and pcmanfm shows an empty folder. Add a slight confusion about the purpose of xdg-open (it does sound like something that opens files, right?), and you get the question you made up.

[-] drkt@scribe.disroot.org 3 points 1 month ago

You can stretch it that far but there doesn't exist a flatpak of pcmanfm anywhere. They'd have to have enough intimate knowledge of Linux and flatpak to build that themselves but then be so stupid as to format a question as poorly as my example?

I should note that it went on to tell me to run some flatpak override commands which I know would break flatpak, so it's definitely making up stuff.

[-] Markaos@discuss.tchncs.de 10 points 1 month ago

But ChatGPT doesn't have a way of "knowing" that there is no such Flatpak - it's unlikely that its training data includes someone explicitly saying that. But it's fair to "assume" that a Linux file manager is available as a Flatpak.

(...), so it's definitely making up stuff.

Yes, it's an LLM

[-] esaru@beehaw.org 3 points 1 month ago

Sure, I added it to the original post above.

[-] Floon@lemmy.ml 15 points 1 month ago

This is not enshittification. It's just shitty.

[-] esaru@beehaw.org 4 points 1 month ago

You are right. I've updated the naming. Thanks for your feedback, very much appreciated.

[-] kehet@sopuli.xyz 11 points 1 month ago* (last edited 1 month ago)

This is not enshittification, this is just a corporation trying to protect itself against anything that could cause negative publicity, like all corporations do. I can even see emojis and positive tone to even be wanted features for some. The real problem here is lack of transparency.

I'm still waiting for ChatGPT etc. to start injecting (more or less hidden) ads to chat and product placement to generated images. That is just unavoidable when bean counters realize that servers and training actually costs money.

[-] esaru@beehaw.org 5 points 1 month ago* (last edited 1 month ago)

OpenAI aims to let users feel better, catering the user's ego, on the costs of reducing the usefulness of the service, rather than getting the message across directly. Their objective is to keep more users on the cost of reducing the utility for the user. It is enshittification in a way, from my point of view.

load more comments (1 replies)
[-] esaru@beehaw.org 8 points 1 month ago* (last edited 1 month ago)

Just to give an impression of how the tone will change after applying the above mentioned custom instructions:

[-] Scipitie@lemmy.dbzer0.com 8 points 1 month ago

Hey,

I'd be very grateful if you could share your approach den if it's only to compare (I went with a "be assertive and clear, skip all overhead" system prompt.

This is not only interesting for chatgpt but understanding how people solve these issues comes in handy when switching to local variants as well!

Thanks in advance

load more comments (1 replies)
[-] gamermanh@lemmy.dbzer0.com 7 points 1 month ago

Sweet fuck am i glad I'm running mine self-hosted and running one of the dolphin models so I can get cool shit like detailed instructions for drug growing and selling or say "fuck" and not have it get angwy at me (tried Gemma and while it's fast... Fucking oof what a locked in corpo AI)

load more comments (1 replies)
load more comments
view more: next ›
this post was submitted on 28 Apr 2025
218 points (100.0% liked)

Technology

38759 readers
318 users here now

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 3 years ago
MODERATORS
alyaza@beehaw.org
TheRtRevKaiser@beehaw.org
gyrfalcon@beehaw.org
rs5th@beehaw.org
coldredlight@beehaw.org
SemioticStandard@beehaw.org
TheRtRevKaiser@kbin.social
remington@beehaw.org