501
We have to stop ignoring AI’s hallucination problem
(www.theverge.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 17 May 2024
501 points (94.8% liked)
Technology
59670 readers
1764 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
A lot of people are developing what I'd call superstitions on some way to overcome LLm limitations. I remember someone swearing they fixed the problem by appending "Ensure the response does not contain hallucinations" to every prompt.
In my experience, what you describe is not a reliable method. Sometimes it's really attached to the same sort of mistakes for the same query. I've seen it double down, when instructed a facet of the answer was incorrect and to revise, several times I'd get "sorry for the incorrect information", followed by exact same mistake. On the flip side, to the extent it "works", it works on valid responses too, meaning an extra pass to ward off "hallucinations" you end up gaslighting the model and it changes the previously correct answer as if it were a hallucination.
How many times are you running it?
For the SelfCheckGPT paper, which was basically this method, it was very sample dependent, continuing to see improvement up to 20 samples (their limit), but especially up to around 6 iterations..
You can't continue with it in context or it ruins the entire methodology. You are reintroducing those tokens when you show it back to the model, and the models are terrible at self-correcting when instructed that it is incorrect, so the step is quite meritless anyways.
You need to run parallel queries and identify shared vs non-shared data points.
It really depends on the specific use case in terms of the full pipeline, but it works really well. Even with just around 5 samples and intermediate summarization steps it pretty much shuts down completely errant hallucinations. The only class of hallucinations it doesn't do great with are the ones resulting from biases in the relationship between the query and the training data, but there's other solutions for things like that.
And yes, it definitely does mean inadvertently eliminating false negatives, which is why a balance has to be struck in terms of design choices.