Access to AI advice significantly degrades human decision-making, causing users to abandon critical judgment while simultaneously inflating their confidence in incorrect answers. A study conducted by researchers at the University of Milano-Bicocca, École Normale Supérieure, and Sapienza University of Rome found that AI assistance caused the willingness to admit ignorance to plummet from 44% to 3%, while user accuracy dropped from 27% to 9%. Despite this performance decline, participant confidence surged from 30% to 76%.
The Erosion of Cognitive Self-Correction
The phenomenon, often described as “cognitive surrender,” occurs when individuals default to AI outputs rather than exercising their own analytical skills. According to Valerio Capraro, an associate professor at the University of Milano-Bicocca, the primary danger lies in the suppression of the human habit of recognizing the limits of one’s own knowledge. The research team deliberately utilized questions where AI models, such as Step 3.5 Flash, are prone to failure—such as specific visual details from films—to test whether users would rely on the tool even when it provided demonstrably incorrect information.
Did you know?
The study found that even when participants were offered monetary incentives for accuracy, their performance remained significantly below the baseline of those working without AI, with accuracy rising only from 9% to 16%.
Design Patterns Driving Over-Reliance
The current architecture of generative AI products exacerbates this trend by design. Most large language models and search-integrated AI tools are built to provide an answer at any cost, rather than admitting a lack of information. Common Sense Media recently labeled the replacement of traditional links with AI-generated summaries an “unacceptable risk,” particularly for students who are still developing critical thinking skills.
Comparing Human vs. AI-Assisted Accuracy
| Metric | No AI | With AI |
|---|---|---|
| Willingness to say “I don’t know” | 44% | 3% |
| Accuracy | 27% | 9% |
| Confidence | 30% | 76% |
Future Risks for Digital Literacy
The findings from Capraro, Chiara Marcoccia, and Walter Quattrociocchi suggest that the mere availability of an AI assistant acts as a cognitive crutch.
Frequently Asked Questions
- What is “cognitive surrender” in the context of AI?
It refers to the tendency for humans to defer to AI-generated answers, often accepting incorrect information while simultaneously reporting higher levels of confidence than they would have without the tool. - Why do people trust wrong AI answers?
Research suggests that the availability of AI suppresses the internal cognitive habit of recognizing what you do not know, leading users to rely on the system’s tone of authority rather than verified facts. - Can monetary incentives fix the accuracy gap?
The study found that while financial rewards slightly improved accuracy (from 9% to 16%), they were insufficient to return performance to the level of users working without AI assistance.
How do you verify the information you receive from AI tools? Share your strategies in the comments below, or subscribe to our newsletter for more updates on the intersection of human psychology and emerging technology.
Related reading