analiza aiqbench
This paper addresses a critical gap in model evaluation by introducing a hierarchical framework that statistically aggregates uncertainty from individual tasks to produce reliable leaderboard-level rank intervals. By combining pairwise comparison confidence intervals with a conformal prediction approach, the method ensures that performance variations across diverse contexts are not obscured, offering a more robust metric for comparing pretrained models. This advancement provides the machine learning community with a principled way to quantify ranking uncertainty, moving beyond single-point estimates that fail to capture task-level variability.
Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation
arXiv:2606.08679v1 Announce Type: cross Abstract: Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts. However, current methods for aggregating performance across tasks into leaderboard-level rankings do not address the uncertainty and variability at the task level. While recent works have proposed interval-based model rankings, the principled aggregation of uncertainty from individual tasks to leaderboard-level rankings remains unaddressed, and variation in models' performance across tasks is frequently obscured. In this work, we introduce a hierarchical framework that constructs model rank intervals with statistical guarantees at both levels: task-level rank confidence intervals from pairwise comparisons, and leaderboard-level rank prediction intervals using a conformal approach. This enables reliable quantification of model rank for each observed task and for new potential tasks. Experiments on simulated data and the TabArena and PromptEval (MMLU) benchmarks show that our method yields statistically valid and informative intervals, enabling reliable, uncertainty-aware model ranking on leaderboards.
Abstract:Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts. However, current methods for aggregating performance across tasks into leaderboard-level rankings do not address the uncertainty and variability at the task level. While recent works have proposed interval-based model rankings, the principled aggregation of uncertainty from individual tasks to leaderboard-level rankings remains unaddressed, and variation in models' performance across tasks is frequently obscured. In this work, we introduce a hierarchical framework that constructs model rank intervals with statistical guarantees at both levels: task-level rank confidence intervals from pairwise comparisons, and leaderboard-level rank prediction intervals using a conformal approach. This enables reliable quantification of model rank for each observed task and for new potential tasks. Experiments on simulated data and the TabArena and PromptEval (MMLU) benchmarks show that our method yields statistically valid and informative intervals, enabling reliable, uncertainty-aware model ranking on leaderboards.
Machine Learning (stat.ML); Computation and Language (cs.CL); Machine Learning (cs.LG); Methodology (stat.ME)
arXiv:2606.08679 [stat.ML]
(or arXiv:2606.08679v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2606.08679 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 250 cuvinte
Subiecte conexe
Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards
arXiv · 30 sept. 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · 4 sept. 2026
Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processes
arXiv · 30 sept. 2026
Rank Confidence Sequences:Anytime-valid Leaderboards
arXiv · 29 sept. 2026