Commit-first LLM judging inherits the judge's own errors

arXiv:2609.00088v1 Announce Type: cross Abstract: LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. Nine implement a variant the literature measures as ineffective, and share one ancestor prompt, traceable through a copied typographical error. In a controlled experiment, an ordinary best-of-N search with no access to correct answers optimises code against one of these configurations, used exactly as documented. On an interval merging task the judge accepted 90 of 96 candidates in one seed and 93 of 96 in the other; every accepted candidate passed every test the search could see and failed a held-out suite it could not. The judge identified the defective line and cited it as grounds for a perfect score. Commit-first judging removed the effect: 0 of 96 in both seeds. On a second task it made matters worse in both seeds: the judge's committed answer was wrong, and in one seed the population converged on it. This is our main finding. Commit-first judging does not remove the anchor that gets gamed, it moves it from the candidate to the judge's own answer, so evaluation is only as good as the judge is at the task. That precondition is cheap to measure in advance, and is task local rather than scale dependent: a smaller judge solved a task the frontier judge failed and resisted gaming where it did not. We also validate our own instruments: five of fifteen claims in our criteria were wrong against verbatim sources, and two held-out checks were unjustified by their specifications.

Abstract:LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. Nine implement a variant the literature measures as ineffective, and share one ancestor prompt, traceable through a copied typographical error. In a controlled experiment, an ordinary best-of-N search with no access to correct answers optimises code against one of these configurations, used exactly as documented. On an interval merging task the judge accepted 90 of 96 candidates in one seed and 93 of 96 in the other; every accepted candidate passed every test the search could see and failed a held-out suite it could not. The judge identified the defective line and cited it as grounds for a perfect score. Commit-first judging removed the effect: 0 of 96 in both seeds. On a second task it made matters worse in both seeds: the judge's committed answer was wrong, and in one seed the population converged on it. This is our main finding. Commit-first judging does not remove the anchor that gets gamed, it moves it from the candidate to the judge's own answer, so evaluation is only as good as the judge is at the task. That precondition is cheap to measure in advance, and is task local rather than scale dependent: a smaller judge solved a task the frontier judge failed and resisted gaming where it did not. We also validate our own instruments: five of fifteen claims in our criteria were wrong against verbatim sources, and two held-out checks were unjustified by their specifications.

Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

arXiv:2609.00088 [cs.SE]

(or arXiv:2609.00088v1 [cs.SE] for this version)

https://doi.org/10.48550/arXiv.2609.00088 Focus to learn more arXiv-issued DOI via DataCite

20260814T112817Z_roman_numerals_as_shipped.jsonl

20260820T134011Z_merge_intervals_as_shipped.jsonl

20260820T134011Z_parse_duration_as_shipped.jsonl

20260820T134011Z_roman_numerals_as_shipped.jsonl

20260820T134011Z_topo_sort_as_shipped.jsonl

20260821T132419Z_merge_intervals_deanchored.jsonl

20260821T132419Z_parse_duration_deanchored.jsonl

20260821T132419Z_roman_numerals_deanchored.jsonl

20260821T132419Z_topo_sort_deanchored.jsonl

20260821T175335Z_merge_intervals_deanchored_opus-4-8.jsonl

20260821T181259Z_parse_duration_deanchored_opus-4-8.jsonl

20260821T185115Z_merge_intervals_as_shipped_strict_opus-4-8.jsonl

20260821T190842Z_merge_intervals_as_shipped_haiku-4-5.jsonl

20260821T192306Z_merge_intervals_deanchored_haiku-4-5.jsonl

20260825T083736Z_merge_intervals_as_shipped_opus-4-8.jsonl

20260825T083736Z_parse_duration_as_shipped_opus-4-8.jsonl

20260825T083736Z_roman_numerals_as_shipped_opus-4-8.jsonl

20260825T083736Z_topo_sort_as_shipped_opus-4-8.jsonl

RESULTS-DIGEST.md

fig1-divergence.csv

fig2-mechanism.csv

fig3-controls.csv

fig4-competence.csv

judge_competence.jsonl

replay-checks.json

replication/README.md

replication/probes.py

replication/replay.py

replication/tasks.py

(25 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Commit-first LLM judging inherits the judge's own errors

Related coverage