aiqbench analysis
This research reveals a critical dissociation in frontier language models where high task accuracy coexists with poor metacognitive confidence, particularly on hard problems or shared errors. The study demonstrates that prompted self-review and peer oversight offer limited improvements for identifying these specific failure modes, highlighting inherent limits in current reliability mechanisms. For developers and safety researchers, this underscores the difficulty of building robust error-detection systems without external reference models or specialized architectures.
On the Limits of Metacognitive Monitoring in LLMs
arXiv:2609.34864v1 Announce Type: new Abstract: Reliable decisions depend on recognizing when an answer may be wrong. In biological cognition, metacognitive monitoring can dissociate from task performance, raising the question of how closely solving and judging are linked in language models. Here we study the confidence reports of four frontier models across 15 benchmarks. High task accuracy can coexist with weak error discrimination: a model solves 97% of competition mathematics problems while its answer-time confidence ranks correct answers above errors barely better than chance. Confidence separates correct answers from errors more effectively on questions solved by a separate reference model, while review brings limited improvement on reference-hard questions. Aggregate discrimination also rewards ranking correct answers on easy questions above errors on hard ones, which question-only forecasts already do well. Cross-evaluation helps most where the evaluator answered correctly, and errors shared by the two models usually retain high confidence. Hard questions and shared errors remain difficult targets for prompted self-review and peer oversight, even in models with strong problem-solving performance.
Abstract:Reliable decisions depend on recognizing when an answer may be wrong. In biological cognition, metacognitive monitoring can dissociate from task performance, raising the question of how closely solving and judging are linked in language models. Here we study the confidence reports of four frontier models across 15 benchmarks. High task accuracy can coexist with weak error discrimination: a model solves 97% of competition mathematics problems while its answer-time confidence ranks correct answers above errors barely better than chance. Confidence separates correct answers from errors more effectively on questions solved by a separate reference model, while review brings limited improvement on reference-hard questions. Aggregate discrimination also rewards ranking correct answers on easy questions above errors on hard ones, which question-only forecasts already do well. Cross-evaluation helps most where the evaluator answered correctly, and errors shared by the two models usually retain high confidence. Hard questions and shared errors remain difficult targets for prompted self-review and peer oversight, even in models with strong problem-solving performance.
Artificial Intelligence (cs.AI); Neurons and Cognition (q-bio.NC)
arXiv:2609.34864 [cs.AI]
(or arXiv:2609.34864v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.34864 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 261 words
Related coverage
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
arXiv · Oct 15, 2026
Implementation of a Metacognition Framework for Self-Awareness and Self-Regulation in Ensembles of LLMs
arXiv · Aug 18, 2026
Self-Modeling Interventions Modulate Emergent Misalignment
LessWrong · Oct 5, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026