aiqbench analysis
This research introduces a rigorous statistical framework for Large Language Model evaluations by quantifying and separating prediction noise from data sampling noise. By proposing an "all-pairs comparison" method that leverages millions of predictions, the study demonstrates how reducing generation variance through averaging can significantly boost the statistical power of benchmarking efforts. The findings provide essential empirical guidance for researchers aiming to design more robust and reliable evaluation protocols in machine learning.
Measuring all the noises of LLM Evals
arXiv:2512.21326v3 Announce Type: replace-cross Abstract: Applying well-established statistical methods effectively to LLM evals requires consideration of model correlations and prediction noises. Leading evals often ignore data noise, have low statistical power, or make unnecessary trade-offs. We define and measure three types of noise: 1) prediction noise from generating different answers on a given question, 2) data noise from sampling questions, and 3) their combined total noise following the law of total variance. To emphasize relative comparisons and gain statistical power, we propose \emph{all-pairs comparison}, which applies paired comparison to all pairs of LLMs and measures all the noise components based on millions of question-level predictions across many evals and settings, revealing clear trends. First, most evals exhibit a characteristic and highly predictable total noise level across all model pairs. Second, paired prediction noise typically exceeds paired data noise, which means reducing prediction noise by averaging can significantly increase statistical power. By measuring all the noises together, we can assess eval results in context and make better empirical decisions.
Abstract:Applying well-established statistical methods effectively to LLM evals requires consideration of model correlations and prediction noises. Leading evals often ignore data noise, have low statistical power, or make unnecessary trade-offs. We define and measure three types of noise: 1) prediction noise from generating different answers on a given question, 2) data noise from sampling questions, and 3) their combined total noise following the law of total variance. To emphasize relative comparisons and gain statistical power, we propose \emph{all-pairs comparison}, which applies paired comparison to all pairs of LLMs and measures all the noise components based on millions of question-level predictions across many evals and settings, revealing clear trends. First, most evals exhibit a characteristic and highly predictable total noise level across all model pairs. Second, paired prediction noise typically exceeds paired data noise, which means reducing prediction noise by averaging can significantly increase statistical power. By measuring all the noises together, we can assess eval results in context and make better empirical decisions.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (stat.ML)
arXiv:2512.21326 [cs.LG]
(or arXiv:2512.21326v3 [cs.LG] for this version)
From: Sida I. Wang [view email] [v1] Wed, 24 Dec 2025 18:54:37 UTC (2,926 KB) [v2] Sun, 29 Mar 2026 00:57:41 UTC (2,938 KB) [v3] Tue, 13 Oct 2026 04:11:28 UTC (3,942 KB)
Full text under the publisher's open licence · 217 words
Related coverage
Why do models task game?
AI Alignment Forum · Aug 6, 2026
Spurious probes as a black-box alternative to activation probing
LessWrong · Sep 25, 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · Jul 21, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026