aiqbench analysis
This study exposes a critical reliability flaw in LLM-as-a-Judge evaluations, defining "dark current" as non-tie preferences between identical inputs that indicate systematic bias rather than genuine judgment. The authors demonstrate that passing standard null gates does not guarantee robust utility performance, revealing significant divergence between aggregate scores and strict subset alignment across different model configurations. These findings underscore the necessity for stratified audit protocols to distinguish between certified reliability and marginal statistical artifacts before deploying judge models in high-stakes scenarios.
LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability
arXiv:2606.15610v2 Announce Type: replace-cross Abstract: Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of .539 and .745 miss the .75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.
Abstract:Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of .539 and .745 miss the .75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.
21 pages, 4 figures, 9 tables. Substantially revised: input-stratified null and target-selection utility audits, additional fixed-configuration studies, and an operational audit protocol. Supersedes v1, originally titled "LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation."
Computation and Language (cs.CL); Instrumentation and Methods for Astrophysics (astro-ph.IM); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2606.15610 [cs.CL]
(or arXiv:2606.15610v2 [cs.CL] for this version)
Full text under the publisher's open licence · 221 words
Related coverage
Personas in a Pre-Internet Language Model
LessWrong · Oct 13, 2026
Does preservation make sense before we know how to revive?
LessWrong · Jun 15, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026