analiza aiqbench
This research introduces a finite-calibration regime map to optimize the selection of LLM judge panels, addressing the critical trade-off between model complexity and limited human labeling budgets. The study demonstrates that scalar or reliability aggregation often outperforms unrestricted joint-table calibration in standard datasets, while more complex interaction models become viable only when sufficient labels are available to eliminate unseen patterns. This work provides a practical framework for evaluating whether additional judge information is estimable under current resource constraints, directly advancing the methodology of automated evaluation systems.
A Finite-Calibration Regime Map for LLM Judge Panels
arXiv:2606.01034v2 Announce Type: replace Abstract: Deploying an LLM judge panel spends human labels on fitting a calibrator, constructing candidate judge paths, and validating which candidate to deploy. We study when finite labels should support a low-dimensional stacker or reliability model, and when an unrestricted joint output table is worth its cell-count and unseen-pattern cost. We cast this as a finite-calibration regime map and instantiate it as Finite-Calibration Panel Selection (FCPS), a validation selector over judge path, deployed panel size, and aggregator family with support diagnostics. Across RewardBench, LLMBar, SummEval, and Arena100K with a seven-judge pool, scalar/reliability aggregation has lower MSE than unrestricted joint-table calibration in 16 of 20 real dataset--budget cells by point estimate, while paired 95% intervals exclude zero in 11 cells; richer backoff/shrinkage tables narrow some gaps while preserving the finite-support bottleneck. Controlled calibration-growth data show the opposite regime: when labels contain a six-way interaction, the selected table grows to the interaction-bearing prefix and its MSE falls from 0.224 to 0.061 once unseen mass vanishes. The practical deployment question is whether the next judge's information is estimable under the available human labels.
Abstract:Deploying an LLM judge panel spends human labels on fitting a calibrator, constructing candidate judge paths, and validating which candidate to deploy. We study when finite labels should support a low-dimensional stacker or reliability model, and when an unrestricted joint output table is worth its cell-count and unseen-pattern cost. We cast this as a finite-calibration regime map and instantiate it as Finite-Calibration Panel Selection (FCPS), a validation selector over judge path, deployed panel size, and aggregator family with support diagnostics. Across RewardBench, LLMBar, SummEval, and Arena100K with a seven-judge pool, scalar/reliability aggregation has lower MSE than unrestricted joint-table calibration in 16 of 20 real dataset--budget cells by point estimate, while paired 95% intervals exclude zero in 11 cells; richer backoff/shrinkage tables narrow some gaps while preserving the finite-support bottleneck. Controlled calibration-growth data show the opposite regime: when labels contain a six-way interaction, the selected table grows to the interaction-bearing prefix and its MSE falls from 0.224 to 0.061 once unseen mass vanishes. The practical deployment question is whether the next judge's information is estimable under the available human labels.
33 pages, 11 figures, 40 tables. Accepted at WISE 2026
Computation and Language (cs.CL); Methodology (stat.ME)
arXiv:2606.01034 [cs.CL]
(or arXiv:2606.01034v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.01034 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 284 cuvinte
Subiecte conexe
Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator
arXiv · 25 aug. 2026
CRC-Screen: Certified DNA-Synthesis Hazard Screening Under Taxonomic Shift
arXiv · 16 iun. 2026
Spectral Anatomy of Quantum Gaussian Process Kernels
arXiv · 3 iun. 2026
Benchtalks #2: The future of coding benchmarks
Snorkel AI Blog · 3 iun. 2026