aiqbench analysis
This research demonstrates that the numerical range assigned to Large Language Model confidence scores significantly impacts their metacognitive sensitivity. By shifting from a standard 0–100 scale to a finer-grained 0–20 format, researchers observed consistent improvements in uncertainty estimation efficiency across multiple models and datasets. The findings underscore that scale design is not merely an implementation detail but a critical variable affecting the reliability of black-box AI evaluations.
Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
arXiv:2603.09309v2 Announce Type: replace Abstract: Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78\% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using $meta\text{-}d'$. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.
Abstract:Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78\% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using $meta\text{-}d'$. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.
Artificial Intelligence (cs.AI)
arXiv:2603.09309 [cs.AI]
(or arXiv:2603.09309v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2603.09309 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 231 words
Related coverage
Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
arXiv · Sep 12, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026
Self-Modeling Interventions Modulate Emergent Misalignment
LessWrong · Oct 5, 2026
Two-dimensional quantum models simulated without full wave function
Quantum Zeitgeist · Sep 9, 2026