Cherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes

Abstract:Evidence-grounded fact verification increasingly uses LLM judges to turn evidence into terminal verdicts. Many task contracts deliberately include non-directional verdicts - Conflicting for materially mixed evidence, Not Enough Evidence for absent evidence - so that a system can decline to assert a direction. Returning Supports or Refutes on a claim whose authorized verdict is non-directional is directional overcommitment: a label error whose output is a direction the contract did not authorize. Its conflict instance - both strands present, one of them returned - is Cherry-pick Override (CPO); unlike the insufficiency instance it cannot be repaired by retrieving more, because nothing was missing. On AVeriTeC's gold-Conflicting subset (N_C=150) a four-option typed panel commits directionally on 18.7% of claims, and four frozen contemporary judges (two proprietary, two open-weight) on 21.3-48.0%, with 24.0-38.0% on VitaminC-Mixed, a constructed stress test on which the ranking reverses. These are conservative lower bounds: the shared prompt specifies Conflicting in detail, forbids collapsing to a direction, and demonstrates the label three times. The failure is not specific to conflict: overcommitment on gold-Insufficient cases runs at a comparable rate for every judge and exceeds CPO for three of four on VitaminC-Mixed. Asked in a fresh context, two judges identify the conflict on about two thirds of their own CPO claims, against under 0.12 where they committed correctly: the recognition is available and does not reach the terminal action. Typed vocabulary, panel voting, confidence thresholds, a structural validator and self-decomposition each leave a substantial residual, and the last buys part of its reduction by relabelling one-sided evidence as conflicting. Every rate is dataset-defined; we report no human-confirmed prevalence.

Substantially revised and extended. The failure covers both non-directional verdicts the contract authorizes, not conflict alone. New probes separate recognition from commitment and bound the rates from below. The planned two-reviewer audit was not completed; all rates stay dataset-defined. 4 judges, 2 substrates. Code: this https URL

Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multiagent Systems (cs.MA)

arXiv:2606.07834 [cs.SE]

(or arXiv:2606.07834v2 [cs.SE] for this version)