analiza aiqbench
This research demonstrates that internal residual stream activations in LLM judges can effectively predict susceptibility to position-order bias, offering a computationally efficient alternative to the resource-intensive double-judgment method. By achieving superior AUROC scores over non-activation baselines without requiring model recalibration, this work provides actionable insights for improving the reliability and fairness of automated evaluation pipelines.
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
arXiv:2610.07115v1 Announce Type: new Abstract: The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.
Title:Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
Abstract:The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.
It was submitted to neurips workshop (JUDGE workshop) and received a review score 6.5 combined with one strong accept and one above threshold accept
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.07115 [cs.LG]
(or arXiv:2610.07115v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.07115 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 271 cuvinte
Subiecte conexe
The basic argument for making AI better at moral philosophy
LessWrong · 14 oct. 2026
OSCAR: Order-aware Scoring and Calibration for AI Rankings
arXiv · 22 sept. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026
MedPAIR: Measuring Whether Physicians and AI Agree on What Matters in Medical QA
Snorkel AI Blog · 2 oct. 2026