CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks

Abstract:An open contrastive decision model is near chance as a judge on the hard public benchmarks: Contrastive-LM/CLM-v0.1-8B scores between 0.351 (best- of-four, chance 0.250) and 0.593 (pairwise, chance 0.500), is statistically indistinguishable from coin flipping on RM-Bench and JudgeBench, and answers every HaluEval item with one constant label, matching the trivial always-first baseline at 0.581. Judges with the same parameter count score far higher everywhere: a reward model reaches 0.764 to 0.976 and a generative judge 0.611 to 0.778, and every gap to CLM is significant after Benjamini-Hochberg correction. Two properties do work. Raw confidences are overconfident by up to +0.401, yet one pooled temperature fit on held-out calibration items repairs expected calibration error to at most 0.062, and the repaired confidence ranks the model's own errors above chance on three of six benchmarks. The decision order-flip rate is 0.0002 against 0.2188 for the generative judge, and the length-preference shift is -0.023 against -0.217. The confidence-gated cascade, however, escalates between 0.923 and 1.000 of items to the strong judge at the preregistered 0.97 retention bar: calibrated confidence about a near-chance judge has almost nothing to keep. The design: five public preference benchmarks and one hallucination benchmark with real labels, scored under a preregistration frozen before any test item was seen, against generative, reward-model, and trivial baselines, with per-item predictions released.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

arXiv:2610.07177 [cs.LG]

(or arXiv:2610.07177v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.07177 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

code/analysis/calibrate.py

code/analysis/gating.py

code/analysis/load.py

code/bootstrap.py

code/build_agreement.py

code/build_appendix_tables.py

code/build_family_table.py

code/build_figures.py

code/build_gate_json.py

code/build_number_check.py

code/build_numbers_tex.py

code/build_splits_release.py

code/build_timeline_tex.py

code/run_analysis.py

code/run_pass.py

manifests/20260927T203410Z_gate02a_card_examples.json

manifests/20260927T203924Z_gate02a_card_examples.json

manifests/20260927T204030Z_gate02c_dual_implementation.json

manifests/20260927T204836Z_gate02c_dual_implementation.json

manifests/20260927T205305Z_gate03_rm_selection.json

manifests/20260927T205726Z_gate04_benchmark_pins.json

manifests/20260927T212410Z_pass_clm_rewardbench_calib_o1.json

manifests/20260927T212510Z_pass_clm_rewardbench2_calib_o1.json

manifests/20260927T212538Z_pass_clm_rewardbench2_pairs_calib_o1.json

manifests/20260927T212557Z_pass_clm_rmbench_calib_o1.json

manifests/20260927T212620Z_pass_clm_judgebench_calib_o1.json

manifests/20260927T212706Z_pass_clm_halueval_calib_o1.json

manifests/20260927T212726Z_pass_laya_rewardbench_calib_o1.json

manifests/20260927T212741Z_pass_laya_rewardbench2_calib_o1.json

manifests/20260927T212757Z_pass_laya_rewardbench2_pairs_calib_o1.json

manifests/20260927T212812Z_pass_laya_rmbench_calib_o1.json

manifests/20260927T212835Z_pass_laya_judgebench_calib_o1.json

manifests/20260927T212910Z_pass_laya_halueval_calib_o1.json

manifests/20260927T212923Z_pass_gen8b_rewardbench_calib_o1.json

manifests/20260927T212957Z_pass_gen8b_rewardbench2_calib_o1.json

manifests/20260927T213042Z_pass_gen8b_rewardbench2_pairs_calib_o1.json

manifests/20260927T213120Z_pass_gen8b_rmbench_calib_o1.json

manifests/20260927T213201Z_pass_gen8b_judgebench_calib_o1.json

manifests/20260927T213259Z_pass_gen8b_halueval_calib_o1.json

manifests/20260927T213411Z_pass_clm_rewardbench2_calib_o1.json

manifests/20260927T213437Z_pass_laya_rewardbench2_calib_o1.json

manifests/20260927T213505Z_pass_gen32b_rewardbench_calib_o1.json

manifests/20260927T213559Z_pass_gen32b_rewardbench2_calib_o1.json

manifests/20260927T213818Z_pass_gen32b_rewardbench2_pairs_calib_o1.json

manifests/20260927T213957Z_pass_gen32b_rmbench_calib_o1.json

manifests/20260927T214058Z_pass_gen32b_judgebench_calib_o1.json

manifests/20260927T214218Z_pass_gen32b_halueval_calib_o1.json

manifests/20260927T214309Z_pass_rm_rewardbench_calib_o1.json

manifests/20260927T214324Z_pass_trivial_always_first_rewardbench_calib_o1.json

manifests/20260927T214332Z_pass_trivial_always_first_rewardbench2_calib_o1.json

manifests/20260927T214341Z_pass_trivial_always_first_rewardbench2_pairs_calib_o1.json

manifests/20260927T214350Z_pass_trivial_always_first_rmbench_calib_o1.json

manifests/20260927T214402Z_pass_trivial_always_first_judgebench_calib_o1.json

manifests/20260927T214423Z_pass_trivial_always_first_halueval_calib_o1.json

manifests/20260927T214431Z_pass_trivial_random_rewardbench_calib_o1.json

manifests/20260927T214439Z_pass_trivial_random_rewardbench2_calib_o1.json

manifests/20260927T214447Z_pass_trivial_random_rewardbench2_pairs_calib_o1.json

manifests/20260927T214455Z_pass_trivial_random_rmbench_calib_o1.json

manifests/20260927T214507Z_pass_trivial_random_judgebench_calib_o1.json

manifests/20260927T214529Z_pass_trivial_random_halueval_calib_o1.json

manifests/20260927T214536Z_pass_trivial_length_rewardbench_calib_o1.json

manifests/20260927T214544Z_pass_trivial_length_rewardbench2_calib_o1.json

manifests/20260927T214553Z_pass_trivial_length_rewardbench2_pairs_calib_o1.json

manifests/20260927T214601Z_pass_trivial_length_rmbench_calib_o1.json

manifests/20260927T214613Z_pass_trivial_length_judgebench_calib_o1.json

manifests/20260927T214634Z_pass_trivial_length_halueval_calib_o1.json

manifests/20260927T214649Z_pass_rm_rewardbench_calib_o1.json

manifests/20260927T214708Z_pass_rm_rewardbench2_calib_o1.json

manifests/20260927T214736Z_pass_rm_rewardbench2_pairs_calib_o1.json

manifests/20260927T214759Z_pass_rm_rmbench_calib_o1.json

manifests/20260927T214821Z_pass_rm_judgebench_calib_o1.json

manifests/20260927T215103Z_gate05_truncation.json

manifests/20260927T215221Z_gate06_heads_data.json

manifests/20260927T215645Z_pass_clm_rewardbench_calib_o1.json

manifests/20260927T220047Z_pass_clm_rewardbench2_calib_o1.json

manifests/20260927T220716Z_pass_clm_rewardbench2_pairs_calib_o1.json

manifests/20260927T221513Z_pass_clm_rmbench_calib_o1.json

manifests/20260927T222312Z_pass_clm_judgebench_calib_o1.json

manifests/20260927T222514Z_pass_clm_halueval_calib_o1.json

manifests/20260927T222756Z_pass_clm_rewardbench_eval_o1.json

manifests/20260927T223748Z_pass_clm_rewardbench2_eval_o1.json

manifests/20260927T225411Z_pass_clm_rewardbench2_pairs_eval_o1.json

manifests/20260927T231318Z_pass_clm_rmbench_eval_o1.json

manifests/20260927T233156Z_pass_clm_judgebench_eval_o1.json

manifests/20260927T233630Z_pass_clm_halueval_eval_o1.json

manifests/20260927T234250Z_pass_clm_rewardbench_eval_o2.json

manifests/20260927T235240Z_pass_clm_rewardbench2_eval_o2.json

manifests/20260928T000904Z_pass_clm_rewardbench2_pairs_eval_o2.json

manifests/20260928T002804Z_pass_clm_rmbench_eval_o2.json

manifests/20260928T004632Z_pass_clm_judgebench_eval_o2.json

manifests/20260928T005052Z_pass_laya_rewardbench_calib_o1.json

manifests/20260928T005124Z_pass_laya_rewardbench2_calib_o1.json

manifests/20260928T005149Z_pass_laya_rewardbench2_pairs_calib_o1.json

manifests/20260928T005234Z_pass_laya_rmbench_calib_o1.json

manifests/20260928T005412Z_pass_laya_judgebench_calib_o1.json

manifests/20260928T005442Z_pass_laya_halueval_calib_o1.json

manifests/20260928T005523Z_pass_laya_rewardbench_eval_o1.json

manifests/20260928T005621Z_pass_laya_rewardbench2_eval_o1.json

manifests/20260928T005706Z_pass_laya_rewardbench2_pairs_eval_o1.json

manifests/20260928T005835Z_pass_laya_rmbench_eval_o1.json

manifests/20260928T010208Z_pass_laya_judgebench_eval_o1.json

manifests/20260928T010246Z_pass_laya_halueval_eval_o1.json

manifests/20260928T010411Z_pass_laya_rewardbench_eval_o2.json

manifests/20260928T010510Z_pass_laya_rewardbench2_eval_o2.json

manifests/20260928T010556Z_pass_laya_rewardbench2_pairs_eval_o2.json

manifests/20260928T010727Z_pass_laya_rmbench_eval_o2.json

manifests/20260928T011108Z_pass_laya_judgebench_eval_o2.json

manifests/20260928T011223Z_train_heads_choice.json

manifests/20260928T011308Z_train_heads_choice.json

manifests/20260928T050138Z_pass_trivial_always_first_rewardbench_eval_o1.json

manifests/20260928T050142Z_pass_trivial_always_first_rewardbench2_eval_o1.json

manifests/20260928T050144Z_pass_trivial_always_first_rewardbench2_pairs_eval_o1.json

manifests/20260928T050147Z_pass_trivial_always_first_rmbench_eval_o1.json

manifests/20260928T050150Z_pass_trivial_always_first_judgebench_eval_o1.json

manifests/20260928T050152Z_pass_trivial_always_first_halueval_eval_o1.json

manifests/20260928T050155Z_pass_trivial_always_first_rewardbench_eval_o2.json

manifests/20260928T050157Z_pass_trivial_always_first_rewardbench2_eval_o2.json

manifests/20260928T050200Z_pass_trivial_always_first_rewardbench2_pairs_eval_o2.json

manifests/20260928T050203Z_pass_trivial_always_first_rmbench_eval_o2.json

manifests/20260928T050205Z_pass_trivial_always_first_judgebench_eval_o2.json

manifests/20260928T050208Z_pass_trivial_random_rewardbench_eval_o1.json

manifests/20260928T050211Z_pass_trivial_random_rewardbench2_eval_o1.json

manifests/20260928T050213Z_pass_trivial_random_rewardbench2_pairs_eval_o1.json

manifests/20260928T050216Z_pass_trivial_random_rmbench_eval_o1.json

manifests/20260928T050219Z_pass_trivial_random_judgebench_eval_o1.json

manifests/20260928T050221Z_pass_trivial_random_halueval_eval_o1.json

manifests/20260928T050224Z_pass_trivial_random_rewardbench_eval_o2.json

manifests/20260928T050227Z_pass_trivial_random_rewardbench2_eval_o2.json

manifests/20260928T050230Z_pass_trivial_random_rewardbench2_pairs_eval_o2.json

manifests/20260928T050232Z_pass_trivial_random_rmbench_eval_o2.json

manifests/20260928T050235Z_pass_trivial_random_judgebench_eval_o2.json

manifests/20260928T050237Z_pass_trivial_length_rewardbench_eval_o1.json

manifests/20260928T050240Z_pass_trivial_length_rewardbench2_eval_o1.json

manifests/20260928T050243Z_pass_trivial_length_rewardbench2_pairs_eval_o1.json

manifests/20260928T050245Z_pass_trivial_length_rmbench_eval_o1.json

manifests/20260928T050248Z_pass_trivial_length_judgebench_eval_o1.json

manifests/20260928T050250Z_pass_trivial_length_halueval_eval_o1.json

manifests/20260928T050253Z_pass_trivial_length_rewardbench_eval_o2.json

manifests/20260928T050255Z_pass_trivial_length_rewardbench2_eval_o2.json

manifests/20260928T050258Z_pass_trivial_length_rewardbench2_pairs_eval_o2.json

manifests/20260928T050301Z_pass_trivial_length_rmbench_eval_o2.json

manifests/20260928T050303Z_pass_trivial_length_judgebench_eval_o2.json

manifests/20260928T050305Z_train_heads_choice.json

manifests/20260928T102302Z_pass_clm_ft_rewardbench_calib_o1.json

manifests/20260928T102736Z_pass_clm_ft_rewardbench2_calib_o1.json

manifests/20260928T103448Z_pass_clm_ft_rewardbench2_pairs_calib_o1.json

manifests/20260928T104246Z_pass_clm_ft_rmbench_calib_o1.json

manifests/20260928T105020Z_pass_clm_ft_judgebench_calib_o1.json

manifests/20260928T105210Z_pass_clm_ft_halueval_calib_o1.json

manifests/20260928T105450Z_pass_clm_ft_rewardbench_eval_o1.json

manifests/20260928T110419Z_pass_clm_ft_rewardbench2_eval_o1.json

manifests/20260928T112040Z_pass_clm_ft_rewardbench2_pairs_eval_o1.json

manifests/20260928T113924Z_pass_clm_ft_rmbench_eval_o1.json

manifests/20260928T115658Z_pass_clm_ft_judgebench_eval_o1.json

manifests/20260928T120104Z_pass_clm_ft_halueval_eval_o1.json

manifests/20260928T120715Z_pass_clm_ft_rewardbench_eval_o2.json

manifests/20260928T121658Z_pass_clm_ft_rewardbench2_eval_o2.json

manifests/20260928T123350Z_pass_clm_ft_rewardbench2_pairs_eval_o2.json

manifests/20260928T125159Z_pass_clm_ft_rmbench_eval_o2.json

manifests/20260928T130927Z_pass_clm_ft_judgebench_eval_o2.json

manifests/20260928T131333Z_pass_gen8b_rewardbench_calib_o1.json

manifests/20260928T132423Z_pass_gen8b_rewardbench2_calib_o1.json

manifests/20260928T133535Z_pass_gen8b_rewardbench2_pairs_calib_o1.json

manifests/20260928T135828Z_pass_gen8b_rmbench_calib_o1.json

manifests/20260928T144921Z_pass_gen8b_judgebench_calib_o1.json

manifests/20260928T145235Z_pass_gen8b_halueval_calib_o1.json

manifests/20260928T150556Z_pass_gen8b_rewardbench_eval_o1.json

manifests/20260928T160405Z_pass_gen8b_rewardbench2_eval_o1.json

manifests/20260928T163032Z_pass_gen8b_rewardbench2_pairs_eval_o1.json

manifests/20260928T173621Z_pass_gen8b_rmbench_eval_o1.json

manifests/20260928T200459Z_pass_gen8b_judgebench_eval_o1.json

manifests/20260928T201432Z_pass_gen8b_halueval_eval_o1.json

manifests/20260928T204919Z_pass_gen8b_rewardbench_eval_o2.json

manifests/20260928T211800Z_pass_gen8b_rewardbench2_eval_o2.json

manifests/20260928T214834Z_pass_gen8b_rewardbench2_pairs_eval_o2.json

manifests/20260928T225219Z_pass_gen8b_rmbench_eval_o2.json

manifests/20260929T011456Z_pass_gen8b_judgebench_eval_o2.json

manifests/20260929T012414Z_pass_rm_rewardbench_calib_o1.json

manifests/20260929T012840Z_pass_rm_rewardbench2_calib_o1.json

manifests/20260929T013745Z_pass_rm_rewardbench2_pairs_calib_o1.json

manifests/20260929T015001Z_pass_rm_rmbench_calib_o1.json

manifests/20260929T021718Z_pass_rm_judgebench_calib_o1.json

manifests/20260929T021939Z_pass_rm_rewardbench_eval_o1.json

manifests/20260929T022936Z_pass_rm_rewardbench2_eval_o1.json

manifests/20260929T024951Z_pass_rm_rewardbench2_pairs_eval_o1.json

manifests/20260929T031854Z_pass_rm_rmbench_eval_o1.json

manifests/20260929T042400Z_pass_rm_judgebench_eval_o1.json

manifests/20260929T042924Z_pass_rm_rewardbench_eval_o2.json

manifests/20260929T050513Z_pass_rm_rewardbench2_eval_o2.json

manifests/20260929T113314Z_pass_gen8b_rewardbench2_calib_o1.json

manifests/20260929T113333Z_pass_gen8b_rewardbench2_pairs_calib_o1.json

manifests/20260929T113413Z_pass_rm_rewardbench2_calib_o1.json

manifests/20260929T113428Z_pass_rm_rewardbench2_pairs_calib_o1.json

manifests/20260929T113451Z_pass_rm_rewardbench2_eval_o2.json

manifests/20260929T114717Z_pass_rm_rewardbench2_pairs_eval_o2.json

manifests/20261005T014951Z_pass_rm_rewardbench2_pairs_eval_o2.json

manifests/20261005T015645Z_pass_rm_rmbench_eval_o2.json

manifests/20261005T030551Z_pass_rm_judgebench_eval_o2.json

manifests/20261005T033136Z_pass_gen32b_rewardbench_calib_o1.json

manifests/20261005T033310Z_pass_gen32b_rewardbench_calib_o1.json

manifests/20261005T034009Z_pass_gen32b_rewardbench2_calib_o1.json

manifests/20261005T034450Z_pass_gen32b_rewardbench2_pairs_calib_o1.json

manifests/20261005T035728Z_pass_gen32b_rmbench_calib_o1.json

manifests/20261005T042706Z_pass_gen32b_judgebench_calib_o1.json

manifests/20261005T042931Z_pass_gen32b_halueval_calib_o1.json

manifests/20261005T043842Z_pass_gen32b_rewardbench_eval_o1.json

manifests/20261005T045430Z_pass_gen32b_rewardbench2_eval_o1.json

manifests/20261005T050535Z_pass_gen32b_rewardbench2_pairs_eval_o1.json

manifests/20261005T053724Z_pass_gen32b_rmbench_eval_o1.json

manifests/20261005T064757Z_pass_gen32b_judgebench_eval_o1.json

manifests/20261005T065216Z_pass_gen32b_halueval_eval_o1.json

manifests/20261005T071632Z_phase3_analysis.json

manifests/20261005T081834Z_phase3_analysis.json

manifests/20261005T101357Z_phase3_analysis.json

metrics/acceptance_gate.json

metrics/benchmarks.json

metrics/bh_summary.json

metrics/calibration_table.json

metrics/cascade_agreement.json

metrics/comparator_table.json

metrics/constants.json

metrics/failure_examples.json

metrics/gate02a_card_examples.json

metrics/gate02c_dual_implementation.json

metrics/gate03_latency.json

metrics/gate05_truncation.json

metrics/gate06_head_training.json

metrics/gate06_heads_data.json

metrics/gating_grid.json

metrics/h10_hallucination.json

metrics/h1_accuracy.json

metrics/h2_calibration.json

metrics/h3_error_detection.json

metrics/h4_cascade.json

metrics/h5_threshold_transfer.json

metrics/h6_gating_rules.json

metrics/h7_position_bias.json

metrics/h8_length_bias.json

metrics/h9_finetuned_heads.json

metrics/hypotheses_bh.json

metrics/rm_selection.json

metrics/subset_accuracy.json

predictions/clm/halueval_calib_order1.jsonl

predictions/clm/halueval_eval_order1.jsonl

predictions/clm/judgebench_calib_order1.jsonl

predictions/clm/judgebench_eval_order1.jsonl

predictions/clm/judgebench_eval_order2.jsonl

predictions/clm/rewardbench2_calib_order1.jsonl

predictions/clm/rewardbench2_eval_order1.jsonl

predictions/clm/rewardbench2_eval_order2.jsonl

predictions/clm/rewardbench2_pairs_calib_order1.jsonl

predictions/clm/rewardbench2_pairs_eval_order1.jsonl

predictions/clm/rewardbench2_pairs_eval_order2.jsonl

predictions/clm/rewardbench_calib_order1.jsonl

predictions/clm/rewardbench_eval_order1.jsonl

predictions/clm/rewardbench_eval_order2.jsonl

predictions/clm/rmbench_calib_order1.jsonl

predictions/clm/rmbench_eval_order1.jsonl

predictions/clm/rmbench_eval_order2.jsonl

predictions/clm_ft/halueval_eval_order1.jsonl

predictions/clm_ft/judgebench_eval_order1.jsonl

predictions/clm_ft/judgebench_eval_order2.jsonl

predictions/clm_ft/rewardbench2_eval_order1.jsonl

predictions/clm_ft/rewardbench2_eval_order2.jsonl

predictions/clm_ft/rewardbench2_pairs_eval_order1.jsonl

predictions/clm_ft/rewardbench2_pairs_eval_order2.jsonl

predictions/clm_ft/rewardbench_eval_order1.jsonl

predictions/clm_ft/rewardbench_eval_order2.jsonl

predictions/clm_ft/rmbench_eval_order1.jsonl

predictions/clm_ft/rmbench_eval_order2.jsonl

predictions/gen32b/halueval_calib_order1.jsonl

predictions/gen32b/halueval_eval_order1.jsonl

predictions/gen32b/judgebench_calib_order1.jsonl

predictions/gen32b/judgebench_eval_order1.jsonl

predictions/gen32b/rewardbench2_calib_order1.jsonl

predictions/gen32b/rewardbench2_eval_order1.jsonl

predictions/gen32b/rewardbench2_pairs_calib_order1.jsonl

predictions/gen32b/rewardbench2_pairs_eval_order1.jsonl

predictions/gen32b/rewardbench_calib_order1.jsonl

predictions/gen32b/rewardbench_eval_order1.jsonl

predictions/gen32b/rmbench_calib_order1.jsonl

predictions/gen32b/rmbench_eval_order1.jsonl

predictions/gen32b_mlx_probes/WHY_MOVED.json

predictions/gen32b_mlx_probes/halueval_calib_order1.jsonl

predictions/gen32b_mlx_probes/judgebench_calib_order1.jsonl

predictions/gen32b_mlx_probes/rewardbench2_calib_order1.jsonl

predictions/gen32b_mlx_probes/rewardbench2_pairs_calib_order1.jsonl

predictions/gen32b_mlx_probes/rewardbench_calib_order1.jsonl

predictions/gen32b_mlx_probes/rmbench_calib_order1.jsonl

predictions/gen8b/halueval_eval_order1.jsonl

predictions/gen8b/judgebench_eval_order1.jsonl

predictions/gen8b/judgebench_eval_order2.jsonl

predictions/gen8b/rewardbench2_eval_order1.jsonl

predictions/gen8b/rewardbench2_eval_order2.jsonl

predictions/gen8b/rewardbench2_pairs_eval_order1.jsonl

predictions/gen8b/rewardbench2_pairs_eval_order2.jsonl

predictions/gen8b/rewardbench_eval_order1.jsonl

predictions/gen8b/rewardbench_eval_order2.jsonl

predictions/gen8b/rmbench_eval_order1.jsonl

predictions/gen8b/rmbench_eval_order2.jsonl

predictions/laya/halueval_eval_order1.jsonl

predictions/laya/judgebench_eval_order1.jsonl

predictions/laya/judgebench_eval_order2.jsonl

predictions/laya/rewardbench2_eval_order1.jsonl

predictions/laya/rewardbench2_eval_order2.jsonl

predictions/laya/rewardbench2_pairs_eval_order1.jsonl

predictions/laya/rewardbench2_pairs_eval_order2.jsonl

predictions/laya/rewardbench_eval_order1.jsonl

predictions/laya/rewardbench_eval_order2.jsonl

predictions/laya/rmbench_eval_order1.jsonl

predictions/laya/rmbench_eval_order2.jsonl

predictions/rm/judgebench_eval_order1.jsonl

predictions/rm/judgebench_eval_order2.jsonl

predictions/rm/rewardbench2_eval_order1.jsonl

predictions/rm/rewardbench2_eval_order2.jsonl

predictions/rm/rewardbench2_pairs_eval_order1.jsonl

predictions/rm/rewardbench2_pairs_eval_order2.jsonl

predictions/rm/rewardbench_eval_order1.jsonl

predictions/rm/rewardbench_eval_order2.jsonl

predictions/rm/rmbench_eval_order1.jsonl

predictions/rm/rmbench_eval_order2.jsonl

predictions/trivial_always_first/halueval_eval_order1.jsonl

predictions/trivial_always_first/judgebench_eval_order1.jsonl

predictions/trivial_always_first/judgebench_eval_order2.jsonl

predictions/trivial_always_first/rewardbench2_eval_order1.jsonl

predictions/trivial_always_first/rewardbench2_eval_order2.jsonl

predictions/trivial_always_first/rewardbench2_pairs_eval_order1.jsonl

predictions/trivial_always_first/rewardbench2_pairs_eval_order2.jsonl

predictions/trivial_always_first/rewardbench_eval_order1.jsonl

predictions/trivial_always_first/rewardbench_eval_order2.jsonl

predictions/trivial_always_first/rmbench_eval_order1.jsonl

predictions/trivial_always_first/rmbench_eval_order2.jsonl

predictions/trivial_length/halueval_eval_order1.jsonl

predictions/trivial_length/judgebench_eval_order1.jsonl

predictions/trivial_length/judgebench_eval_order2.jsonl

predictions/trivial_length/rewardbench2_eval_order1.jsonl

predictions/trivial_length/rewardbench2_eval_order2.jsonl

predictions/trivial_length/rewardbench2_pairs_eval_order1.jsonl

predictions/trivial_length/rewardbench2_pairs_eval_order2.jsonl

predictions/trivial_length/rewardbench_eval_order1.jsonl

predictions/trivial_length/rewardbench_eval_order2.jsonl

predictions/trivial_length/rmbench_eval_order1.jsonl

predictions/trivial_length/rmbench_eval_order2.jsonl

predictions/trivial_random/halueval_eval_order1.jsonl

predictions/trivial_random/judgebench_eval_order1.jsonl

predictions/trivial_random/judgebench_eval_order2.jsonl

predictions/trivial_random/rewardbench2_eval_order1.jsonl

predictions/trivial_random/rewardbench2_eval_order2.jsonl

predictions/trivial_random/rewardbench2_pairs_eval_order1.jsonl

predictions/trivial_random/rewardbench2_pairs_eval_order2.jsonl

predictions/trivial_random/rewardbench_eval_order1.jsonl

predictions/trivial_random/rewardbench_eval_order2.jsonl

predictions/trivial_random/rmbench_eval_order1.jsonl

predictions/trivial_random/rmbench_eval_order2.jsonl

prereg_freeze.json

preregistration.md

splits/halueval.json

splits/judgebench.json

splits/rewardbench.json

splits/rewardbench2.json

splits/rewardbench2_pairs.json

splits/rmbench.json

(363 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.