aiqbench analysis
This research challenges the assumption that high-confidence errors in large language models indicate fragile internal inference, proposing instead that they may stem from stable miscalibration where wrong answers remain locally robust to perturbations. By combining output-level audit scores with hidden-state sensitivity probes across multiple domains and open-weight models, the study demonstrates how self-critical prompting can reduce decision loss without necessarily improving calibration or reducing local stability of errors. These findings offer a nuanced diagnostic framework for understanding model reliability that moves beyond simple confidence metrics toward analyzing internal state dynamics under forced-answer baselines.
Stable Miscalibration in Large Language Models: A Practical View of High-Confidence Errors
arXiv:2608.13591v1 Announce Type: cross Abstract: High-confidence errors in large language models are often treated as evidence of fragile internal inference. We study a different possibility: stable miscalibration, where a confident wrong answer remains locally stable under small perturbations. We combine two diagnostics: a label-aware output-level audit score that ranks domains by confidence variation and overconfident mistakes under a forced-answer baseline, and an internal sensitivity probe that measures hidden-state movement. On a multi-domain binary factual audit set, this audit score tracks where abstention-aware self-critique reduces decision loss, although direct labeled baselines rank the same gain more strongly. Internally, self-critical prompting consistently reduces hidden-state sensitivity across layers in three open-weight models. This supports prompt-induced local stabilization rather than a purely output-level abstention pattern, but it does not imply calibration: audit-defined overconfident errors are not clearly more locally sensitive than confidently correct answers, so some high-confidence errors may be stable and miscalibrated rather than simply fragile.
Abstract:High-confidence errors in large language models are often treated as evidence of fragile internal inference. We study a different possibility: stable miscalibration, where a confident wrong answer remains locally stable under small perturbations. We combine two diagnostics: a label-aware output-level audit score that ranks domains by confidence variation and overconfident mistakes under a forced-answer baseline, and an internal sensitivity probe that measures hidden-state movement. On a multi-domain binary factual audit set, this audit score tracks where abstention-aware self-critique reduces decision loss, although direct labeled baselines rank the same gain more strongly. Internally, self-critical prompting consistently reduces hidden-state sensitivity across layers in three open-weight models. This supports prompt-induced local stabilization rather than a purely output-level abstention pattern, but it does not imply calibration: audit-defined overconfident errors are not clearly more locally sensitive than confidently correct answers, so some high-confidence errors may be stable and miscalibrated rather than simply fragile.
Accepted at the 2nd Workshop on Epistemic Intelligence in Machine Learning (EIML)@ICML 2026. 7 pages, 4 figures
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2608.13591 [cs.AI]
(or arXiv:2608.13591v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.13591 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 261 words
Related coverage
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026