Swapping the Judge Changes the Ranking: A Preregistered Study of Cross-Provider Replication in LLM-Judged System Comparisons

Abstract:Practitioners who compare systems with an LLM judge and have no human labels often check a conclusion by repeating it with a judge from another provider. We test how far that check can be trusted. In a preregistered study (with a disclosed post-registration reduction of the judge set), five judge models from three providers re-judged 4,000 Chatbot Arena battles (nine systems, 33 system pairs) that carry human votes, each in both response orders. Agreement with the human vote was low at the verdict level (median Cohen's kappa = 0.22), and system rankings did not survive a change of judge: Kendall's tau with the human ranking ranged from 0.17 to 0.89 (median 0.67), so the preregistered hypothesis that rankings remain stable was not supported. Conclusions replicated by two low-cost judges from different providers matched the human conclusion more often than single-judge conclusions (+0.10 precision), but the gain came from filtering out one poorly aligned judge, and replication involving that judge left a conclusion for only 35% of comparisons. Judges that agreed with humans reached a stable conclusion with no more verdicts than human votes required; the others failed to converge on 26-70% of system pairs at the sample sizes tested (up to 100 verdicts per pair). Disagreements with the human vote co-occurred across judges far more often than independence predicts, although a single human vote per battle cannot separate shared judge bias from voter noise.

10 pages, 2 figures. Code and judge verdicts are provided as ancillary files

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)

arXiv:2611.04090 [cs.CL]

(or arXiv:2611.04090v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2611.04090 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

judgerep_release/ANONYMIZATION.md

judgerep_release/CHANGES_POST_AUDIT.md

judgerep_release/DEVIATIONS.md

judgerep_release/DEVIATIONS_EN.md

judgerep_release/FIVE_JUDGE_DEVIATION.json

judgerep_release/FREEZE.json

judgerep_release/PREREGISTRATION.md

judgerep_release/PREREGISTRATION_EN.md

judgerep_release/README.md

judgerep_release/REPRODUCTION_CHECK.json

judgerep_release/RESEARCH_DEVIATION_2026-10-06.md

judgerep_release/bundle_manifest.csv

judgerep_release/config.yaml

judgerep_release/data/derived/battles.parquet

judgerep_release/data/derived/cost_ledger.json

judgerep_release/data/derived/parsed.parquet

judgerep_release/data/derived/verdicts.parquet

judgerep_release/outputs/analysis.json

judgerep_release/outputs/figures/h1_kappa_tau.pdf

judgerep_release/outputs/figures/h2_precision.pdf

judgerep_release/outputs/figures/h3_sample_curves.pdf

judgerep_release/outputs/figures/h4_error_correlations.pdf

judgerep_release/outputs/human_power_check.json

judgerep_release/outputs/paper_check/FINAL_SUMMARY.md

judgerep_release/outputs/paper_check/bt_ranking_long.csv

judgerep_release/outputs/paper_check/bt_rankings.csv

judgerep_release/outputs/paper_check/build_report.json

judgerep_release/outputs/paper_check/change_timeline.csv

judgerep_release/outputs/paper_check/curated_facts.csv

judgerep_release/outputs/paper_check/execution_settings.csv

judgerep_release/outputs/paper_check/filtering.csv

judgerep_release/outputs/paper_check/final_edits.diff

judgerep_release/outputs/paper_check/final_validation.json

judgerep_release/outputs/paper_check/kappa_matrix.csv

judgerep_release/outputs/paper_check/kappa_orientation_check.csv

judgerep_release/outputs/paper_check/language.csv

judgerep_release/outputs/paper_check/methods_audit.csv

judgerep_release/outputs/paper_check/null_tau_check.csv

judgerep_release/outputs/paper_check/number_audit.csv

judgerep_release/outputs/paper_check/number_audit_manifest.json

judgerep_release/outputs/paper_check/number_mismatches.csv

judgerep_release/outputs/paper_check/number_unresolved.csv

judgerep_release/outputs/paper_check/official_model_name.md

judgerep_release/outputs/paper_check/provenance_bank.csv

judgerep_release/outputs/paper_check/rank_distance.csv

judgerep_release/outputs/paper_check/table1.csv

judgerep_release/outputs/paper_check/table2.csv

judgerep_release/outputs/paper_check/table3.csv

judgerep_release/outputs/paper_check/table4.csv

judgerep_release/outputs/posthoc/h1_without_oai.csv

judgerep_release/outputs/posthoc/h2_delta_by_combination.csv

judgerep_release/outputs/posthoc/h4_by_human_gap.csv

judgerep_release/outputs/posthoc/human_significant_wrong_pairs.csv

judgerep_release/outputs/posthoc/integrity_checks.csv

judgerep_release/outputs/posthoc/kappa_matrix.csv

judgerep_release/outputs/posthoc/kappa_matrix_n.csv

judgerep_release/outputs/posthoc/linkage/status.json

judgerep_release/outputs/posthoc/oai_dev_confirm.csv

judgerep_release/outputs/posthoc/pair_comparison_33.csv

judgerep_release/outputs/posthoc/position_bias.csv

judgerep_release/outputs/posthoc/shared_wrong_pairs.csv

judgerep_release/outputs/posthoc/wrong_pairs_by_judge.csv

judgerep_release/outputs/tables/battle_accuracy.csv

judgerep_release/outputs/tables/bt_strengths.csv

judgerep_release/outputs/tables/budget_balance.csv

judgerep_release/outputs/tables/confirm_budget_projection.csv

judgerep_release/outputs/tables/confirm_cost_actual.csv

judgerep_release/outputs/tables/confirm_token_projection.csv

judgerep_release/outputs/tables/cost.csv

judgerep_release/outputs/tables/coverage.csv

judgerep_release/outputs/tables/dev_execution_status.csv

judgerep_release/outputs/tables/dry_run_budget_groups.csv

judgerep_release/outputs/tables/dry_run_costs.csv

judgerep_release/outputs/tables/error_correlations.csv

judgerep_release/outputs/tables/execution_quality.csv

judgerep_release/outputs/tables/filter_counts.csv

judgerep_release/outputs/tables/h1.csv

judgerep_release/outputs/tables/h1_language.csv

judgerep_release/outputs/tables/h2_precision.csv

judgerep_release/outputs/tables/human_power_check.csv

judgerep_release/outputs/tables/hypotheses.csv

judgerep_release/outputs/tables/judge_agreement.csv

judgerep_release/outputs/tables/judge_summary.csv

judgerep_release/outputs/tables/language_distribution.csv

judgerep_release/outputs/tables/nstar.csv

judgerep_release/outputs/tables/nstar_curves.csv

judgerep_release/outputs/tables/nstar_distribution.csv

judgerep_release/outputs/tables/nstar_ratios.csv

judgerep_release/outputs/tables/null_comparison.csv

judgerep_release/outputs/tables/null_model.csv

judgerep_release/outputs/tables/pair_counts_available.csv

judgerep_release/outputs/tables/pair_counts_sampled.csv

judgerep_release/outputs/tables/pair_decisions.csv

judgerep_release/outputs/tables/pair_size_distribution.csv

judgerep_release/outputs/tables/parse_quality.csv

judgerep_release/outputs/tables/prompt_hash_counts.csv

judgerep_release/outputs/tables/prompt_language_audit.csv

judgerep_release/outputs/tables/prompt_repetition_top.csv

judgerep_release/outputs/tables/provider_exclusion_audit.csv

judgerep_release/outputs/tables/provider_preference.csv

judgerep_release/outputs/tables/provider_preference_contrast.csv

judgerep_release/outputs/tables/quality_rates.csv

judgerep_release/outputs/tables/refused_overlap.csv

judgerep_release/outputs/tables/repeat_reliability.csv

judgerep_release/outputs/tables/sampling_attempts.csv

judgerep_release/outputs/tables/schema_columns.csv

judgerep_release/outputs/tables/schema_conversation_a_lengths.csv

judgerep_release/outputs/tables/schema_conversation_b_lengths.csv

judgerep_release/outputs/tables/schema_evaluation_orders.csv

judgerep_release/outputs/tables/schema_full_conversation_lengths.csv

judgerep_release/outputs/tables/schema_languages.csv

judgerep_release/outputs/tables/schema_metadata_turns.csv

judgerep_release/outputs/tables/schema_system_providers.csv

judgerep_release/outputs/tables/schema_systems.csv

judgerep_release/outputs/tables/schema_votes.csv

judgerep_release/outputs/tables/self_exclusion_supplementary.csv

judgerep_release/outputs/tables/self_preference.csv

judgerep_release/outputs/tables/system_providers.csv

judgerep_release/outputs/tables/unknown_providers.csv

judgerep_release/outputs/tables/unknown_system_providers.csv

judgerep_release/paper/acl.sty

judgerep_release/paper/acl_natbib.bst

judgerep_release/paper/figures/h1_kappa_tau.pdf

judgerep_release/paper/figures/h2_precision.pdf

judgerep_release/paper/figures/h3_sample_curves.pdf

judgerep_release/paper/figures/h4_error_correlations.pdf

judgerep_release/paper/main.tex

judgerep_release/paper/refs.bib

judgerep_release/paper/tables/README.md

judgerep_release/paper/tables/bt_rankings.tex

judgerep_release/paper/tables/execution_settings.tex

judgerep_release/paper/tables/filtering.tex

judgerep_release/paper/tables/kappa_matrix.tex

judgerep_release/paper/tables/language.tex

judgerep_release/paper/tables/table1.tex

judgerep_release/paper/tables/table2.tex

judgerep_release/paper/tables/table3.tex

judgerep_release/paper/tables/table4.tex

judgerep_release/pyproject.toml

judgerep_release/requirements.lock

judgerep_release/src/judgerep/__init__.py

judgerep_release/src/judgerep/__main__.py

judgerep_release/src/judgerep/adapters.py

judgerep_release/src/judgerep/analysis.py

judgerep_release/src/judgerep/aux/__init__.py

judgerep_release/src/judgerep/aux/justrank.py

judgerep_release/src/judgerep/aux/paper.py

judgerep_release/src/judgerep/aux/posthoc.py

judgerep_release/src/judgerep/baselines/__init__.py

judgerep_release/src/judgerep/baselines/ppi.py

judgerep_release/src/judgerep/cli.py

judgerep_release/src/judgerep/config.py

judgerep_release/src/judgerep/cost.py

judgerep_release/src/judgerep/credentials.py

judgerep_release/src/judgerep/data.py

judgerep_release/src/judgerep/demo.py

judgerep_release/src/judgerep/errors.py

judgerep_release/src/judgerep/freeze.py

judgerep_release/src/judgerep/human_power.py

judgerep_release/src/judgerep/parse.py

judgerep_release/src/judgerep/partial_analysis.py

judgerep_release/src/judgerep/prompts.py

judgerep_release/src/judgerep/provider_preference.py

judgerep_release/src/judgerep/quality.py

judgerep_release/src/judgerep/registration.py

judgerep_release/src/judgerep/report.py

judgerep_release/src/judgerep/requests.py

judgerep_release/src/judgerep/resolve.py

judgerep_release/src/judgerep/runner.py

judgerep_release/src/judgerep/sample_audit.py

judgerep_release/src/judgerep/sampling.py

judgerep_release/src/judgerep/stats/__init__.py

judgerep_release/src/judgerep/stats/agreement.py

judgerep_release/src/judgerep/stats/bt.py

judgerep_release/src/judgerep/stats/common_error.py

judgerep_release/src/judgerep/stats/decisions.py

judgerep_release/src/judgerep/stats/nstar.py

judgerep_release/src/judgerep/stats/null_model.py

judgerep_release/src/judgerep/storage.py

judgerep_release/tests/conftest.py

judgerep_release/tests/test_cli.py

judgerep_release/tests/test_config.py

judgerep_release/tests/test_end_to_end.py

judgerep_release/tests/test_execution.py

judgerep_release/tests/test_five_judges.py

judgerep_release/tests/test_inspect.py

judgerep_release/tests/test_justrank.py

judgerep_release/tests/test_preregistration.py

judgerep_release/tests/test_provider_preference.py

judgerep_release/tests/test_research.py

judgerep_release/tests/test_sample_audit.py

(188 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.