aiqbench analysis
This study applies Signal Detection Theory to disentangle LLM accuracy from metacognitive confidence, revealing that calibration metrics often fail to capture model-specific signal structures invisible to standard error rates. The findings demonstrate that an AI's ability to gauge its own uncertainty is not linearly correlated with factual correctness and varies significantly by domain and temperature settings. These insights are critical for developing more robust evaluation frameworks that distinguish between what a model knows and how reliably it can report on that knowledge.
Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory
arXiv:2603.25112v3 Announce Type: replace-cross Abstract: Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate two capacities: how much a model knows (Type-1 accuracy) and how well its confidence signal tracks that knowledge (Type-2 metacognitive sensitivity). We apply Signal Detection Theory to decompose them, treating token-level normalised log-probability as a graded confidence variable and answer correctness as the state to be discriminated. We characterise the Type-2 ROC of this signal, including its unequal-variance structure via z-ROC analysis, and -- because the meta-d' efficiency ratio is not well defined for open-ended QA, which lacks a two-alternative Type-1 decision -- quantify efficiency with a model-free information measure, normalised metacognitive information (meta-I_2r). Across four LLMs and 224,000 factual QA trials we find: (1) metacognitive information varies by a factor of 1.98 across models and is not predicted by accuracy, the rank correlation being -0.80 on TriviaQA and +0.00 on Natural Questions; (2) the confidence signal has model-specific unequal-variance structure (z-ROC slopes 0.78 to 1.18) invisible to calibration metrics, the slope ordering replicating on NQ; (3) efficiency is domain-specific, weakest in Science & Technology for every model; (4) temperature dissociates accuracy from metacognitive information, which stays near-flat for three of four models while accuracy falls; and (5) metacognitive information tracks the accuracy gain from confidence-based abstention exactly (rho = +1.00) while accuracy does not. All estimates carry permutation nulls and bootstrap confidence intervals. This version (v3) corrects a differential length bias in the automated correctness scorer, validated against 1,830 human adjudications; the inverse accuracy-efficiency coupling reported in v1 and v2 does not survive relabelling. See the version note on page 1.
Abstract:Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate two capacities: how much a model knows (Type-1 accuracy) and how well its confidence signal tracks that knowledge (Type-2 metacognitive sensitivity). We apply Signal Detection Theory to decompose them, treating token-level normalised log-probability as a graded confidence variable and answer correctness as the state to be discriminated. We characterise the Type-2 ROC of this signal, including its unequal-variance structure via z-ROC analysis, and -- because the meta-d' efficiency ratio is not well defined for open-ended QA, which lacks a two-alternative Type-1 decision -- quantify efficiency with a model-free information measure, normalised metacognitive information (meta-I_2r). Across four LLMs and 224,000 factual QA trials we find: (1) metacognitive information varies by a factor of 1.98 across models and is not predicted by accuracy, the rank correlation being -0.80 on TriviaQA and +0.00 on Natural Questions; (2) the confidence signal has model-specific unequal-variance structure (z-ROC slopes 0.78 to 1.18) invisible to calibration metrics, the slope ordering replicating on NQ; (3) efficiency is domain-specific, weakest in Science & Technology for every model; (4) temperature dissociates accuracy from metacognitive information, which stays near-flat for three of four models while accuracy falls; and (5) metacognitive information tracks the accuracy gain from confidence-based abstention exactly (rho = +1.00) while accuracy does not. All estimates carry permutation nulls and bootstrap confidence intervals. This version (v3) corrects a differential length bias in the automated correctness scorer, validated against 1,830 human adjudications; the inverse accuracy-efficiency coupling reported in v1 and v2 does not survive relabelling. See the version note on page 1.
12 pages, 4 figures, 3 tables. v3 corrects a differential length bias in the automated correctness scorer, validated against 1,830 human adjudications; all analyses recomputed. The inverse accuracy-efficiency coupling in v1/v2 does not survive relabelling; see the version note on page 1. Pre-registered. Code and annotations: this http URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2603.25112 [cs.CL]
(or arXiv:2603.25112v3 [cs.CL] for this version)
Full text under the publisher's open licence · 330 words
Related coverage
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · Sep 14, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026