Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap

arXiv:2610.15778v1 Announce Type: cross Abstract: When a policy is trained with GRPO against a frozen, reference-free LLM judge, the judge accepts more of its answers, but in a post-hoc analysis we detect no gain in held-out greedy accuracy: in all 12 (policy, judge) cells, the seed-mean gain never exceeded 0.04. A gold-reward control with the same recipe raised the 1.5B policy's greedy accuracy by +0.054 [0.013, 0.096], so for this policy the flat accuracy is specific to the judge reward; for the 0.5B policy, gold reward also gave no detectable gain. In the four rising cells we checked, the early rise in training-rollout accuracy is mostly sampling sharpening, as it is under gold reward. We then ask whether the policy-judge gap or the judge sets how far and how fast RL exploits the judge's false positives. We train 0.5B and 1.5B Qwen2.5 policies against four judges at least as large (Qwen2.5-1.5B/3B/7B, Llama-3.2-3B) and the 1.5B policy against one smaller judge (Qwen3-0.6B). The depth of judge hacking (the plateau) tracks the judge: it correlates with judge AUC (Spearman -0.69 [-0.94, -0.14] over 12 cells), and within a family the larger judge is fooled less deeply. For the pace (onset), two pre-specified pairs contradict the gap account (the larger-gap cell was fooled 3.5x sooner at the pre-specified threshold). With 12 cells we cannot formally separate judge AUC from the gap for either endpoint. The smaller judge was not fooled sooner than a judge as large as the policy, though it also differs in family and template. In a blind test on three held-out judges, a hash-locked step-0 predictor did not beat step-0 AUC, but three cells cannot separate them. At 7-8B, judges from three families plateaued at 0.37, 0.67 and 0.85.

Title:Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap

Abstract:Under GRPO against a frozen, reference-free LLM judge, RL exploits the judge's false positives. Does the policy-judge capability gap or the judge itself set how fast and how far? We train 0.5B and 1.5B Qwen2.5 policies on GSM8K against four judges at least as large, and the 1.5B policy also against one smaller judge. In a post-hoc analysis, judge acceptance rose, but no cell of 12 gained more than 0.04 in seed-mean held-out greedy accuracy. A gold-reward control with the same recipe raised the 1.5B policy's accuracy by +0.054 (for the 0.5B policy, gold reward gave no detectable gain either). The depth of judge hacking tracks the judge: the plateau correlates with judge AUC (Spearman -0.69 over 12 cells), and within a family the larger judge is fooled less deeply. For the pace, two pre-specified pairs contradict the gap account: the cell with the larger gap was fooled 3.5x sooner at the pre-specified threshold. With 12 cells, however, we cannot formally separate judge AUC from the gap for either endpoint. A blind test of step-0 predictors on three held-out judges was inconclusive. At 7-8B, judges from three families plateaued between 0.37 and 0.85.

20 pages, 7 figures. v2: shorter abstract, revised AI-use disclosure, layout fixes. Code, per-step logs and the read answer samples: this https URL

Machine Learning (cs.LG); Computation and Language (cs.CL)

arXiv:2610.15778 [cs.LG]

(or arXiv:2610.15778v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.15778 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap

Subiecte conexe