AI Advice Makes People Less Accurate but Twice as Confident
AI advice made people 3x less accurate but 2x confident, researchers found
New research reveals that relying on AI advice causes people to become three times less accurate while doubling their confidence. The study shows that access to AI suppresses the willingness to admit ignorance, leading users to trust wrong answers from models like Step 3.5 Flash. Even with monetary incentives, accuracy remains low, highlighting a dangerous trend of cognitive surrender where critical thinking skills are eroded by over-reliance on automated systems.
For humans, the capacity to say 'I don't know' is very important because it represents the recognition of the limits of our own knowledge.
- dwohnitmok
This study is pretty bad. The comment (https://news.ycombinator.com/item?id=48970182) on the other link with the direct PDF explains the problem well, which is that nothing here being tested is specific to AI systems.
This study gave people access to an LLM that the researchers knew would give incorrect answers to certain questions, and then quizzed people on those questions, with the option to not respond to a given question if they are unsure about the answer.
This is akin to giving someone a textbook on an obscure subject that has certain factual errors, letting them know they can use that textbook in a quiz on that subject, and then quizzing that person on those facts that the textbook gets wrong.
Obviously that person is both more likely to be willing to respond to the question and is more likely to get it wrong!
There are a lot of things I'm very interested in that are specific to modern LLMs and how they affect learning and confidence (sycophancy, cognitive helplessness, etc.).
This study tested none of those. Its experimental setup is not very different than simply substituting the LLM with a textbook with errors.
- reticulates
Advice and information subreddits have gone to shit because of AI usage. A large number of people seem to think that when someone asks a question, what they really want is not someone with direct knowledge, but instead someone to relay the question to ChatGPT and post the result as if it is their own hard earned knowledge and insight. I have no idea about the quality of this research but in the real world (well, real-ish, as real as Reddit can be) it is stark, people aren’t just refusing to say “I don’t know” they’re actively seeking out opportunities to pretend they know things.
- wisty
AI pessimist view.
Even if AI gets smarter it will still be agreeable, and people will use it to more confidently reinforce thier stupider ideas, especially in areas where they lack the knowledge to know if they are right.
Even if this can be solved, technically, people won't want to use the model that says they are wrong, so they will choose the glib lies that reinforce their beliefs.
Freedown of speech forces people to think they may be wrong, but freedom of association lets people avoid this. AI is going to be internet hug box echo chambers at an unbelievable scale.
- senshan
This could serve as a beacon for those who still care:
Richard Feynman: "I have the advantage of having found out how hard it is to get to really know something, how careful you have to be about checking the experiments, how easy it is to make mistakes and fool yourself..."
- andy99
This was posted earlier but didn’t get traction, and I made the following comment:
Id want to know if “AI” makes a material difference vs just having access to the wrong answer. Like someone could be given search access that successfully retrieved wrong answers to questions, would that give the same results. How much do uniquely AI characteristics, like sycophancy or the conversational aspect play into this, vs people just being willing to believe what they read?
- leecommamichael
I read the study. The wager is:
$0.10 awarded for a correct answer
$0.10 deducted for a wrong answer
$0.00 (no change for a refusal to answer)
There was no detail on whether participants were actually going to be paid, or if they were in the negative at the end, be expected to pay their losses.
I am inclined to believe the effects of the study are real, but not nearly as pronounced as the data. If there were more serious amounts of money on the table, I think common sense would prevail.
- rho138
I appreciate that they’re bringing this study to light… however, none of the source links actually trace back to the quoted study - instead pointing to their own domain or business insider.
- imoverclocked
I saw a post that some teachers are now asking students to ask ChatGPT to do their assignments… and then critique it where it is wrong.
I feel like this might be an interesting method for places where the default response tends to be someone submitting copy-pasta: just give the ai-response and then have community discussion around what it missed.