Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Abstract:Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels. Yet the same verdicts can support wildly varying agreement numbers, depending on seemingly minor choices: the judgment scale, the retained cases, the handling of abstentions and invalid outputs, and the pooling of verdicts across items and rubric criteria. The statistics that settle these choices are established, but in psychometrics, econometrics, and corpus annotation rather than in the evaluation practice that needs them. We treat the choices as a measurement protocol that fixes what the reported number estimates before any metric is computed, assemble the relevant results into a single source-attributed analysis, and apply it to three published LLM-judge evaluations. For non-degenerate binary verdicts, Pearson's $r$, Spearman's $\rho$, Kendall's $\tau_b$, the phi coefficient, and the Matthews correlation coefficient are exactly the same statistic, so reporting several repeats one number under different names. Cohen's $\kappa$ differs from them only through a marginal-mismatch factor in $(0,1]$ and shares their asymptotic variance when judge and human assign the positive verdict equally often. Under exclusion, accuracy over all cases is pinned down only to a worst-case interval as wide as the uncovered fraction. On a rubric benchmark carrying per-criterion human labels, protocol choice alone moves reported accuracy from $0.551$ to $0.899$ and carries $\kappa$ across zero, without altering a single verdict. We distill the analysis into a reporting checklist that makes agreement claims reconstructible and comparable.

Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Data Analysis, Statistics and Probability (physics.data-an)

arXiv:2606.00093 [cs.CL]

(or arXiv:2606.00093v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.00093 Focus to learn more arXiv-issued DOI via DataCite

data/cascade_splits.csv

data/llm_rubric/LICENSE_LLM-Rubric.txt

data/llm_rubric/gpt-3.5-turbo-16k_real_evaluations_FIXED.tsv

data/llm_rubric/human_judges_real_convs_FIXED_ANON.tsv

data/scan_coding.csv

reproduce_cascade.py

reproduce_examples.py

reproduce_rubric.py

reproduce_selfpref.py

sweep_cascade_splits.py

(8 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.