analiza aiqbench
JudgeMoE addresses the loss of uncertainty information inherent in scalar compression by aggregating cached score distributions with example-specific weights, offering a measurable improvement over uniform log pooling on standard benchmarks. The study highlights that optimal aggregation strategies are task-dependent and generally favor soft scoring methods over hard decoding to preserve nuanced evaluation data. This work advances the reliability of LLM-as-a-Judge frameworks by refining how multiple model outputs are fused for final scoring.
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
arXiv:2610.07109v1 Announce Type: new Abstract: When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge--dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by $+0.079$. Applying the same configuration to six additional cells yields a $+0.0393$ mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank $p=0.0091$. Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.
Abstract:When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge--dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by $+0.079$. Applying the same configuration to six additional cells yields a $+0.0393$ mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank $p=0.0091$. Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.
Computation and Language (cs.CL)
arXiv:2610.07109 [cs.CL]
(or arXiv:2610.07109v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.07109 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 224 cuvinte
Subiecte conexe
Always ask your agent to be honest
LessWrong · 16 sept. 2026
Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge
arXiv · 3 iun. 2026
Deconfusing Corrigibility
AI Alignment Forum · 17 oct. 2026
Post-hoc Alignment of LLM-judges to Human Judgment Distribution
arXiv · 2 sept. 2026