aiqbench analysis
This research extends Item Response Theory to handle continuous scoring metrics like ROUGE and LLM-as-a-Judge, enabling efficient model ranking with significantly fewer evaluation items. By introducing an uncertainty-aware ranker, the method achieves high-confidence predictions using only 2% of standard test sets while maintaining strong correlation over random sampling. This approach directly addresses the cost-efficiency needs of modern LLM development workflows where generation tasks dominate evaluation pipelines.
Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
arXiv:2601.13885v2 Announce Type: replace-cross Abstract: Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method improves ranking correlation by 0.13 $\tau$ over random sampling and has 99% accuracy on confident predictions while using 2% of the items after a one-time calibration step.
Abstract:Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method improves ranking correlation by 0.13 $\tau$ over random sampling and has 99% accuracy on confident predictions while using 2% of the items after a one-time calibration step.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2601.13885 [cs.CL]
(or arXiv:2601.13885v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2601.13885 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 225 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Build an explainable next-best-product recommendation system for banking on AWS
AWS · Jul 24, 2026
[AINews] not much happened today
Latent Space · Oct 3, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026