AI Advice Makes People Less Accurate but Twice as Confident
AI advice made people 3x less accurate but 2x confident, researchers found
New research reveals that relying on AI advice causes people to become three times less accurate while doubling their confidence. The study shows that access to AI suppresses the willingness to admit ignorance, leading users to trust wrong answers from models like Step 3.5 Flash. Even with monetary incentives, accuracy remains low, highlighting a dangerous trend of cognitive surrender where critical thinking skills are eroded by over-reliance on automated systems.
"For humans, the capacity to say 'I don't know' is very important because it represents the recognition of the limits of our own knowledge."
HN discussion
- The study's design is flawed because it compares AI to a Google search without accounting for the sycophantic tone and framing that make AI answers uniquely persuasive compared to raw search results.
- The core issue is not the AI's accuracy but the human tendency to trust well-written text in a chat interface over their own verification, especially when they prioritize saving time over fact-checking.
- The headline misrepresents the study, which deliberately examines how people rely on inaccurate data when AI is insufficient, a critical risk in complex fields like healthcare where patients provide less context than doctors.
- Unlike textbooks with known factual errors, users often treat LLMs as authoritative sources despite knowing they hallucinate, leading to a dangerous overconfidence that differs from traditional information consumption.
- Recent testing shows that prompting models like Gemini and ChatGPT with 'are you sure?' can trigger self-correction, whereas other models like GLM 5 Turbo remain confidently incorrect even when challenged.