aiqbench analysis
This research demonstrates that the numerical range of confidence scales in LLMs is not neutral, showing that a 0–20 scale significantly improves metacognitive efficiency compared to the standard 0–100 format. The study reveals that model outputs are heavily discretized around round numbers, a bias that degrades performance when boundary compression occurs but persists regardless of range regularity. These findings establish confidence scale design as a critical experimental variable for accurately evaluating uncertainty in large language models.
Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
arXiv:2603.09309v3 Announce Type: replace Abstract: Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78\% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using $meta\text{-}d'$. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.
Abstract:Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78\% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using $meta\text{-}d'$. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.
Artificial Intelligence (cs.AI)
arXiv:2603.09309 [cs.AI]
(or arXiv:2603.09309v3 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2603.09309 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 231 words
Related coverage
Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
arXiv · Jun 16, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026
Self-Modeling Interventions Modulate Emergent Misalignment
LessWrong · Oct 5, 2026
Two-dimensional quantum models simulated without full wave function
Quantum Zeitgeist · Sep 9, 2026