JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

At our latest Snorkel AI Reading Group, Russell Yang (AI Engineering Fellow at Stanford Law) stopped by our San Francisco office to present JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment. As AI models improve at open-ended tasks, the field faces a harder problem: how to measure quality in domains where ground truth is contested. Two paradigms dominate: rubric-based scoring, where annotators apply predefined criteria to a single output, and comparative judgment, where annotators select the holistically better of two outputs side by side. Both are widely used, but the choice between them is rarely empirically justified, especially in domains like law, consulting, and finance where expert judgment is the only available signal. JudgmentBench is the first publicly available dataset to elicit both supervision signals from the same experts on the same items: 30 real-world legal tasks annotated by 50+ practicing attorneys from major U.S. law firms, yielding more than 3,000 paired rubric scores and pairwise preference judgments built on top of Harvey’s BigLawBench. The results are striking: comparative judgments recover intended quality ordering nearly perfectly (mean Spearman’s ρ = 0.908) while rubrics barely beat chance (ρ = 0.150), and pairwise judgments require less than half the annotation time,…