aiqbench analysis
This research introduces a judge-aware ranking framework that accounts for varying reliability among LLM judges, addressing a critical flaw in current evaluation paradigms where equal weighting leads to biased leaderboards. By extending the Bradley-Terry-Luce model with judge-specific parameters, the method provides statistically grounded confidence intervals and improves alignment with human preferences without requiring ground-truth labels. For practitioners building evaluation pipelines, this approach offers a more robust and data-efficient way to quantify model performance and uncertainty in open-ended tasks.
A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
arXiv:2601.21817v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates. More data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.
Abstract:Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates. More data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2601.21817 [stat.ML]
(or arXiv:2601.21817v3 [stat.ML] for this version)
Full text under the publisher's open licence · 156 words