Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?

Title:Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?

Abstract:Reinforcement learning with verifiable rewards (RLVR) is credited with gains beyond the domain its rewards check, such as higher MMLU accuracy after training on mathematics. Such accuracies are usually read out of the model's text by an answer extractor. A model that learns to write its answer where the extractor looks can therefore score higher without answering better. We measure how much of the gain reported by extractors copied from public code arises this way. We generate outputs from a base model and its RL model and score them twice: with those extractors, and with one extractor of our own that reads a stated answer whether it is in the answer box \boxed{} or in the text. On four released 7B SimpleRL base-RL pairs, 82% of the out-of-domain multiple-choice gain that these extractors report disappears under ours: it is a gain in answer format, not in answers. The base models already state most of their answers in text, and RL moves them into the box, where these extractors look first. Independent LLM readers, in place of our extractor, put the figure at 77%. With a prompt that itself asks for the box, our extractor puts it at 79%. On a 32B pair trained with another recipe, almost none of the reported gain is format gain: its base model boxes its answers when asked, and its RL model rarely does when not. A reported gain is partly a property of the extractor that read it, and re-scoring stored outputs with a second extractor shows how much, without re-running RL.

Machine Learning (cs.LG); Computation and Language (cs.CL)

arXiv:2610.20639 [cs.LG]

(or arXiv:2610.20639v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.20639 Focus to learn more arXiv-issued DOI via DataCite

data/items/MANIFEST.json

data/items/aqua_rat.jsonl

data/items/arc_c.jsonl

data/items/bbh.jsonl

data/items/gsm8k.jsonl

data/items/math500.jsonl

data/items/mmlu.jsonl

data/items/mmlu_pro.jsonl

data/items/sat_math.jsonl

data/llama-pilot-items/arc_c.jsonl

data/llama-pilot-items/mmlu.jsonl

data/llama-pilot-items/mmlu_pro.jsonl

harness/adjudicate.py

harness/adjudication_descriptive.py

harness/adjudication_prompt.txt

harness/adjudication_result.py

harness/analyze.py

harness/answer_tags.py

harness/boxing_by_prompt.py

harness/build_generation_manifest.py

harness/correctness_criterion.py

harness/decompose.py

harness/denominator_weight.py

harness/derive_supporting.py

harness/dose_response.py

harness/dose_response_pooled.py

harness/external_matched.py

harness/external_reference.py

harness/extraction_rate_settings.py

harness/extractors.py

harness/first_pilot_generate.py

harness/format_compliance.py

harness/freeze_items.py

harness/gen_model.py

harness/gen_unit.py

harness/larger_pairs.py

harness/llama_pilot.py

harness/llama_pilot_arc_c.py

harness/make_figures_2_3.py

harness/make_figures_4_5.py

harness/math_equiv.py

harness/mechanism_by_prompt.py

harness/mechanism_main_pairs.py

harness/omega_matched.py

harness/omega_sensitivity.py

harness/parser_departures.py

harness/parser_fidelity.py

harness/parser_reach_matrix.py

harness/parser_reach_nonempty.py

harness/poison_test.py

harness/pooled.py

harness/publish_bounds.py

harness/published_vs_reread.py

harness/qwen_repair_impact.py

harness/qwen_strip_string.py

harness/release_loop.py

harness/release_loop_effect.py

harness/running_example.py

harness/running_example_misses.py

harness/share_checks.py

harness/shipped_parsers.py

harness/transcript_differential.py

harness/verify_upstream_pins.py

harness/with_letter_step.py

records/adjudication-precommitment.CORRECTION.md

records/adjudication-precommitment.json

records/first-pilot/abstaining_extractor.py

records/first-pilot/accuracy_where_read.py

records/first-pilot/extraction_rate_criterion.py

records/first-pilot/extractors_on_grid.py

records/first-pilot/format_share.py

records/first-pilot/generations/gen-base.json

records/first-pilot/generations/gen-post.json

records/first-pilot/outputs_without_answer.py

records/first-pilot/permissive_extractor.py

records/first-pilot/prompts.py

records/first-pilot/reconciliation.json

records/first-pilot/result.json

records/first-pilot/results/accuracy-where-read.json

records/first-pilot/results/extraction-rate-criterion.json

records/first-pilot/results/extractors-on-grid.json

records/first-pilot/results/format-share.json

records/first-pilot/results/outputs-without-answer.json

records/first-pilot/results/prompts.json

records/harness-checks/model-roster.json

records/harness-checks/parser-fidelity.json

records/harness-checks/poison-test.json

records/harness-checks/transcript-differential.json

records/larger-pairs-precommitment.json

records/release-survey.json

records/screening/parsers-transcribed.json

records/screening/rlvr-pairs-verified.json

records/upstream-pin-verification.json

requirements.txt

results/MANIFEST.json

results/MANIFEST.md

results/adjudication-descriptive.json

results/adjudication-result.json

results/adjudication/RESULT.md

results/adjudication/batches.json

results/adjudication/human-kit/README.md

results/adjudication/human-kit/answers-to-fill.csv

results/adjudication/human-kit/items.html

results/adjudication/raw/first/0000.attempt0.json

results/adjudication/raw/first/0001.attempt0.json

results/adjudication/raw/first/0002.attempt0.json

results/adjudication/raw/first/0003.attempt0.json

results/adjudication/raw/first/0004.attempt0.json

results/adjudication/raw/first/0005.attempt0.json

results/adjudication/raw/first/0006.attempt0.json

results/adjudication/raw/first/0007.attempt0.json

results/adjudication/raw/first/0008.attempt0.json

results/adjudication/raw/first/0009.attempt0.json

results/adjudication/raw/first/0010.attempt0.json

results/adjudication/raw/first/0011.attempt0.json

results/adjudication/raw/first/0012.attempt0.json

results/adjudication/raw/first/0013.attempt0.json

results/adjudication/raw/first/0014.attempt0.json

results/adjudication/raw/first/0015.attempt0.json

results/adjudication/raw/first/0016.attempt0.json

results/adjudication/raw/first/0017.attempt0.json

results/adjudication/raw/first/0018.attempt0.json

results/adjudication/raw/first/0019.attempt0.json

results/adjudication/raw/first/0020.attempt0.json

results/adjudication/raw/first/0021.attempt0.json

results/adjudication/raw/first/0022.attempt0.json

results/adjudication/raw/first/0023.attempt0.json

results/adjudication/raw/first/0024.attempt0.json

results/adjudication/raw/first/0025.attempt0.json

results/adjudication/raw/first/0026.attempt0.json

results/adjudication/raw/first/0027.attempt0.json

results/adjudication/raw/first/0028.attempt0.json

results/adjudication/raw/first/0029.attempt0.json

results/adjudication/raw/first/0030.attempt0.json

results/adjudication/raw/first/0031.attempt0.json

results/adjudication/raw/first/0032.attempt0.json

results/adjudication/raw/first/0033.attempt0.json

results/adjudication/raw/first/0034.attempt0.json

results/adjudication/raw/first/0035.attempt0.json

results/adjudication/raw/first/0036.attempt0.json

results/adjudication/raw/first/0037.attempt0.json

results/adjudication/raw/first/0038.attempt0.json

results/adjudication/raw/first/0039.attempt0.json

results/adjudication/raw/first/0040.attempt0.json

results/adjudication/raw/first/0041.attempt0.json

results/adjudication/raw/first/0042.attempt0.json

results/adjudication/raw/first/0043.attempt0.json

results/adjudication/raw/first/0044.attempt0.json

results/adjudication/raw/first/0045.attempt0.json

results/adjudication/raw/first/0046.attempt0.json

results/adjudication/raw/first/0047.attempt0.json

results/adjudication/raw/first/0048.attempt0.json

results/adjudication/raw/first/0049.attempt0.json

results/adjudication/raw/first/0050.attempt0.json

results/adjudication/raw/first/0051.attempt0.json

results/adjudication/raw/first/0052.attempt0.json

results/adjudication/raw/first/0053.attempt0.json

results/adjudication/raw/first/0054.attempt0.json

results/adjudication/raw/first/0055.attempt0.json

results/adjudication/raw/first/0056.attempt0.json

results/adjudication/raw/first/0057.attempt0.json

results/adjudication/raw/first/0058.attempt0.json

results/adjudication/raw/first/0059.attempt0.json

results/adjudication/raw/first/0060.attempt0.json

results/adjudication/raw/first/0061.attempt0.json

results/adjudication/raw/first/0062.attempt0.json

results/adjudication/raw/first/0063.attempt0.json

results/adjudication/raw/first/0064.attempt0.json

results/adjudication/raw/first/0065.attempt0.json

results/adjudication/raw/first/0066.attempt0.json

results/adjudication/raw/first/0067.attempt0.json

results/adjudication/raw/first/0068.attempt0.json

results/adjudication/raw/first/0069.attempt0.json

results/adjudication/raw/first/0070.attempt0.json

results/adjudication/raw/first/0071.attempt0.json

results/adjudication/raw/first/0072.attempt0.json

results/adjudication/raw/first/0073.attempt0.json

results/adjudication/raw/first/0074.attempt0.json

results/adjudication/raw/first/0075.attempt0.json

results/adjudication/raw/first/0076.attempt0.json

results/adjudication/raw/first/0077.attempt0.json

results/adjudication/raw/first/0078.attempt0.json

results/adjudication/raw/first/0079.attempt0.json

results/adjudication/raw/first/0080.attempt0.json

results/adjudication/raw/first/0081.attempt0.json

results/adjudication/raw/first/0082.attempt0.json

results/adjudication/raw/first/0083.attempt0.json

results/adjudication/raw/first/0084.attempt0.json

results/adjudication/raw/first/0085.attempt0.json

results/adjudication/raw/first/0086.attempt0.json

results/adjudication/raw/first/0087.attempt0.json

results/adjudication/raw/first/0088.attempt0.json

results/adjudication/raw/first/0089.attempt0.json

results/adjudication/raw/first/0090.attempt0.json

results/adjudication/raw/first/0091.attempt0.json

results/adjudication/raw/first/0092.attempt0.json

results/adjudication/raw/first/0093.attempt0.json

results/adjudication/raw/first/0094.attempt0.json

results/adjudication/raw/first/0095.attempt0.json

results/adjudication/raw/first/0096.attempt0.json

results/adjudication/raw/first/0097.attempt0.json

results/adjudication/raw/first/0098.attempt0.json

results/adjudication/raw/first/0099.attempt0.json

results/adjudication/raw/first/0100.attempt0.json

results/adjudication/raw/first/0101.attempt0.json

results/adjudication/raw/first/0102.attempt0.json

results/adjudication/raw/first/0103.attempt0.json

results/adjudication/raw/first/0104.attempt0.json

results/adjudication/raw/first/0105.attempt0.json

results/adjudication/raw/first/0106.attempt0.json

results/adjudication/raw/first/0107.attempt0.json

results/adjudication/raw/first/0108.attempt0.json

results/adjudication/raw/first/0109.attempt0.json

results/adjudication/raw/first/0110.attempt0.json

results/adjudication/raw/first/0111.attempt0.json

results/adjudication/raw/first/0112.attempt0.json

results/adjudication/raw/first/0113.attempt0.json

results/adjudication/raw/first/0114.attempt0.json

results/adjudication/raw/first/0114.attempt1.json

results/adjudication/raw/first/0115.attempt0.json

results/adjudication/raw/first/0116.attempt0.json

results/adjudication/raw/first/0117.attempt0.json

results/adjudication/raw/first/0118.attempt0.json

results/adjudication/raw/first/0119.attempt0.json

results/adjudication/raw/first/0120.attempt0.json

results/adjudication/raw/first/0121.attempt0.json

results/adjudication/raw/first/0122.attempt0.json

results/adjudication/raw/first/0123.attempt0.json

results/adjudication/raw/first/0124.attempt0.json

results/adjudication/raw/first/0125.attempt0.json

results/adjudication/raw/first/0126.attempt0.json

results/adjudication/raw/first/0127.attempt0.json

results/adjudication/raw/first/0128.attempt0.json

results/adjudication/raw/first/0129.attempt0.json

results/adjudication/raw/first/0130.attempt0.json

results/adjudication/raw/first/0131.attempt0.json

results/adjudication/raw/first/0132.attempt0.json

results/adjudication/raw/first/0133.attempt0.json

results/adjudication/raw/first/0134.attempt0.json

results/adjudication/raw/first/0135.attempt0.json

results/adjudication/raw/first/0136.attempt0.json

results/adjudication/raw/first/0137.attempt0.json

results/adjudication/raw/first/0137.attempt1.json

results/adjudication/raw/first/0138.attempt0.json

results/adjudication/raw/first/0139.attempt0.json

results/adjudication/raw/first/0140.attempt0.json

results/adjudication/raw/first/0141.attempt0.json

results/adjudication/raw/first/0142.attempt0.json

results/adjudication/raw/first/0143.attempt0.json

results/adjudication/raw/first/0143.attempt1.json

results/adjudication/raw/first/0144.attempt0.json

results/adjudication/raw/first/0145.attempt0.json

results/adjudication/raw/first/0146.attempt0.json

results/adjudication/raw/first/0147.attempt0.json

results/adjudication/raw/first/0148.attempt0.json

results/adjudication/raw/first/0149.attempt0.json

results/adjudication/raw/first/0150.attempt0.json

results/adjudication/raw/first/0151.attempt0.json

results/adjudication/raw/first/0152.attempt0.json

results/adjudication/raw/first/0153.attempt0.json

results/adjudication/raw/first/0154.attempt0.json

results/adjudication/raw/first/0155.attempt0.json

results/adjudication/raw/first/0156.attempt0.json

results/adjudication/raw/first/0157.attempt0.json

results/adjudication/raw/first/0158.attempt0.json

results/adjudication/raw/first/0159.attempt0.json

results/adjudication/raw/first/0160.attempt0.json

results/adjudication/raw/first/0161.attempt0.json

results/adjudication/raw/first/0162.attempt0.json

results/adjudication/raw/first/0163.attempt0.json

results/adjudication/raw/first/0164.attempt0.json

results/adjudication/raw/first/0165.attempt0.json

results/adjudication/raw/first/0166.attempt0.json

results/adjudication/raw/first/0167.attempt0.json

results/adjudication/raw/first/0168.attempt0.json

results/adjudication/raw/first/0169.attempt0.json

results/adjudication/raw/first/0170.attempt0.json

results/adjudication/raw/first/0171.attempt0.json

results/adjudication/raw/first/0172.attempt0.json

results/adjudication/raw/first/0173.attempt0.json

results/adjudication/raw/first/0174.attempt0.json

results/adjudication/raw/first/0175.attempt0.json

results/adjudication/raw/first/0176.attempt0.json

results/adjudication/raw/first/0177.attempt0.json

results/adjudication/raw/first/0178.attempt0.json

results/adjudication/raw/first/0179.attempt0.json

results/adjudication/raw/first/0180.attempt0.json

results/adjudication/raw/first/0181.attempt0.json

results/adjudication/raw/first/0182.attempt0.json

results/adjudication/raw/first/0183.attempt0.json

results/adjudication/raw/first/0184.attempt0.json

results/adjudication/raw/first/0185.attempt0.json

results/adjudication/raw/first/0186.attempt0.json

results/adjudication/raw/first/0187.attempt0.json

results/adjudication/raw/first/0188.attempt0.json

results/adjudication/raw/first/0189.attempt0.json

results/adjudication/raw/first/0190.attempt0.json

results/adjudication/raw/first/0191.attempt0.json

results/adjudication/raw/first/0192.attempt0.json

results/adjudication/raw/first/0193.attempt0.json

results/adjudication/raw/first/0194.attempt0.json

results/adjudication/raw/first/0195.attempt0.json

results/adjudication/raw/first/0196.attempt0.json

results/adjudication/raw/first/0197.attempt0.json

results/adjudication/raw/first/0198.attempt0.json

results/adjudication/raw/first/0199.attempt0.json

results/adjudication/raw/first/0200.attempt0.json

results/adjudication/raw/first/0201.attempt0.json

results/adjudication/raw/first/0202.attempt0.json

results/adjudication/raw/first/0203.attempt0.json

results/adjudication/raw/first/0204.attempt0.json

results/adjudication/raw/first/0205.attempt0.json

results/adjudication/raw/first/0206.attempt0.json

results/adjudication/raw/first/0206.attempt1.json

results/adjudication/raw/first/0207.attempt0.json

results/adjudication/raw/first/0208.attempt0.json

results/adjudication/raw/first/0209.attempt0.json

results/adjudication/raw/first/0210.attempt0.json

results/adjudication/raw/first/0211.attempt0.json

results/adjudication/raw/first/0212.attempt0.json

results/adjudication/raw/first/0213.attempt0.json

results/adjudication/raw/first/0214.attempt0.json

results/adjudication/raw/first/0215.attempt0.json

results/adjudication/raw/first/0216.attempt0.json

results/adjudication/raw/first/0217.attempt0.json

results/adjudication/raw/first/0218.attempt0.json

results/adjudication/raw/first/0219.attempt0.json

results/adjudication/raw/first/0220.attempt0.json

results/adjudication/raw/first/0221.attempt0.json

results/adjudication/raw/first/0222.attempt0.json

results/adjudication/raw/first/0223.attempt0.json

results/adjudication/raw/first/0224.attempt0.json

results/adjudication/raw/first/0225.attempt0.json

results/adjudication/raw/first/0226.attempt0.json

results/adjudication/raw/first/0227.attempt0.json

results/adjudication/raw/first/0228.attempt0.json

results/adjudication/raw/first/0229.attempt0.json

results/adjudication/raw/first/0230.attempt0.json

results/adjudication/raw/first/0231.attempt0.json

results/adjudication/raw/first/0231.attempt1.json

results/adjudication/raw/first/0232.attempt0.json

results/adjudication/raw/first/0233.attempt0.json

results/adjudication/raw/first/0234.attempt0.json

results/adjudication/raw/first/0235.attempt0.json

results/adjudication/raw/first/0236.attempt0.json

results/adjudication/raw/first/0237.attempt0.json

results/adjudication/raw/first/0238.attempt0.json

results/adjudication/raw/first/0239.attempt0.json

results/adjudication/raw/first/0240.attempt0.json

results/adjudication/raw/first/0241.attempt0.json

results/adjudication/raw/first/0242.attempt0.json

results/adjudication/raw/first/0243.attempt0.json

results/adjudication/raw/first/0244.attempt0.json

results/adjudication/raw/first/0245.attempt0.json

results/adjudication/raw/first/0246.attempt0.json

results/adjudication/raw/first/0247.attempt0.json

results/adjudication/raw/first/0248.attempt0.json

results/adjudication/raw/first/0249.attempt0.json

results/adjudication/raw/first/0250.attempt0.json

results/adjudication/raw/first/0251.attempt0.json

results/adjudication/raw/first/0252.attempt0.json

results/adjudication/raw/first/0253.attempt0.json

results/adjudication/raw/first/0254.attempt0.json

results/adjudication/raw/first/0255.attempt0.json

results/adjudication/raw/first/0256.attempt0.json

results/adjudication/raw/first/0257.attempt0.json

results/adjudication/raw/first/0258.attempt0.json

results/adjudication/raw/first/0259.attempt0.json

results/adjudication/raw/first/0260.attempt0.json

results/adjudication/raw/first/0261.attempt0.json

results/adjudication/raw/first/0262.attempt0.json

results/adjudication/raw/first/0263.attempt0.json

results/adjudication/raw/first/0264.attempt0.json

results/adjudication/raw/first/0265.attempt0.json

results/adjudication/raw/first/0266.attempt0.json

results/adjudication/raw/first/0267.attempt0.json

results/adjudication/raw/first/0268.attempt0.json

results/adjudication/raw/first/0269.attempt0.json

results/adjudication/raw/first/0270.attempt0.json

results/adjudication/raw/first/0271.attempt0.json

results/adjudication/raw/second/0000.attempt0.json

results/adjudication/raw/second/0001.attempt0.json

results/adjudication/raw/second/0002.attempt0.json

results/adjudication/raw/second/0003.attempt0.json

results/adjudication/raw/second/0004.attempt0.json

results/adjudication/raw/second/0005.attempt0.json

results/adjudication/raw/second/0006.attempt0.json

results/adjudication/raw/second/0007.attempt0.json

results/adjudication/raw/second/0008.attempt0.json

results/adjudication/raw/second/0009.attempt0.json

results/adjudication/raw/second/0010.attempt0.json

results/adjudication/raw/second/0011.attempt0.json

results/adjudication/raw/second/0012.attempt0.json

results/adjudication/raw/second/0013.attempt0.json

results/adjudication/raw/second/0014.attempt0.json

results/adjudication/raw/second/0015.attempt0.json

results/adjudication/raw/second/0016.attempt0.json

results/adjudication/raw/second/0017.attempt0.json

results/adjudication/raw/second/0018.attempt0.json

results/adjudication/raw/second/0019.attempt0.json

results/adjudication/raw/second/0020.attempt0.json

results/adjudication/raw/second/0021.attempt0.json

results/adjudication/raw/second/0022.attempt0.json

results/adjudication/raw/second/0023.attempt0.json

results/adjudication/raw/second/0024.attempt0.json

results/adjudication/raw/second/0025.attempt0.json

results/adjudication/raw/second/0026.attempt0.json

results/adjudication/raw/second/0027.attempt0.json

results/adjudication/raw/second/0028.attempt0.json

results/adjudication/raw/second/0029.attempt0.json

results/adjudication/raw/second/0030.attempt0.json

results/adjudication/raw/second/0031.attempt0.json

results/adjudication/raw/second/0032.attempt0.json

results/adjudication/raw/second/0033.attempt0.json

results/adjudication/raw/second/0034.attempt0.json

results/adjudication/raw/second/0035.attempt0.json

results/adjudication/raw/second/0036.attempt0.json

results/adjudication/raw/second/0037.attempt0.json

results/adjudication/raw/second/0038.attempt0.json

results/adjudication/raw/second/0039.attempt0.json

results/adjudication/raw/second/0040.attempt0.json

results/adjudication/raw/second/0041.attempt0.json

results/adjudication/raw/second/0042.attempt0.json

results/adjudication/raw/second/0043.attempt0.json

results/adjudication/raw/second/0044.attempt0.json

results/adjudication/raw/second/0045.attempt0.json

results/adjudication/raw/second/0046.attempt0.json

results/adjudication/raw/second/0047.attempt0.json

results/adjudication/raw/second/0048.attempt0.json

results/adjudication/raw/second/0049.attempt0.json

results/adjudication/raw/second/0050.attempt0.json

results/adjudication/raw/second/0051.attempt0.json

results/adjudication/raw/second/0052.attempt0.json

results/adjudication/raw/second/0053.attempt0.json

results/adjudication/raw/second/0054.attempt0.json

results/adjudication/raw/second/0055.attempt0.json

results/adjudication/raw/second/0056.attempt0.json

results/adjudication/raw/second/0057.attempt0.json

results/adjudication/raw/second/0058.attempt0.json

results/adjudication/raw/second/0059.attempt0.json

results/adjudication/raw/second/0060.attempt0.json

results/adjudication/raw/second/0061.attempt0.json

results/adjudication/raw/second/0062.attempt0.json

results/adjudication/raw/second/0063.attempt0.json

results/adjudication/raw/second/0064.attempt0.json

results/adjudication/raw/second/0065.attempt0.json

results/adjudication/raw/second/0066.attempt0.json

results/adjudication/raw/second/0067.attempt0.json

results/adjudication/raw/second/0068.attempt0.json

results/adjudication/raw/second/0069.attempt0.json

results/adjudication/raw/second/0070.attempt0.json

results/adjudication/raw/second/0071.attempt0.json

results/adjudication/raw/second/0072.attempt0.json

results/adjudication/raw/second/0073.attempt0.json

results/adjudication/raw/second/0074.attempt0.json

results/adjudication/raw/second/0075.attempt0.json

results/adjudication/raw/second/0076.attempt0.json

results/adjudication/raw/second/0077.attempt0.json

results/adjudication/raw/second/0078.attempt0.json

results/adjudication/raw/second/0079.attempt0.json

results/adjudication/raw/second/0080.attempt0.json

results/adjudication/raw/second/0081.attempt0.json

results/adjudication/raw/second/0082.attempt0.json

results/adjudication/raw/second/0083.attempt0.json

results/adjudication/raw/second/0084.attempt0.json

results/adjudication/raw/second/0085.attempt0.json

results/adjudication/raw/second/0086.attempt0.json

results/adjudication/raw/second/0087.attempt0.json

results/adjudication/raw/second/0088.attempt0.json

results/adjudication/raw/second/0089.attempt0.json

results/adjudication/raw/second/0090.attempt0.json

results/adjudication/raw/second/0091.attempt0.json

results/adjudication/raw/second/0092.attempt0.json

results/adjudication/raw/second/0093.attempt0.json

results/adjudication/raw/second/0094.attempt0.json

results/adjudication/raw/second/0095.attempt0.json

results/adjudication/raw/second/0096.attempt0.json

results/adjudication/raw/second/0097.attempt0.json

results/adjudication/raw/second/0098.attempt0.json

results/adjudication/raw/second/0099.attempt0.json

results/adjudication/raw/second/0100.attempt0.json

results/adjudication/raw/second/0101.attempt0.json

results/adjudication/raw/second/0102.attempt0.json

results/adjudication/raw/second/0103.attempt0.json

results/adjudication/raw/second/0104.attempt0.json

results/adjudication/raw/second/0105.attempt0.json

results/adjudication/raw/second/0106.attempt0.json

results/adjudication/raw/second/0107.attempt0.json

results/adjudication/raw/second/0108.attempt0.json

results/adjudication/raw/second/0109.attempt0.json

results/adjudication/raw/second/0110.attempt0.json

results/adjudication/raw/second/0111.attempt0.json

results/adjudication/raw/second/0112.attempt0.json

results/adjudication/raw/second/0113.attempt0.json

results/adjudication/raw/second/0114.attempt0.json

results/adjudication/raw/second/0115.attempt0.json

results/adjudication/raw/second/0116.attempt0.json

results/adjudication/raw/second/0117.attempt0.json

results/adjudication/raw/second/0118.attempt0.json

results/adjudication/raw/second/0119.attempt0.json

results/adjudication/raw/second/0120.attempt0.json

results/adjudication/raw/second/0121.attempt0.json

results/adjudication/raw/second/0122.attempt0.json

results/adjudication/raw/second/0123.attempt0.json

results/adjudication/raw/second/0124.attempt0.json

results/adjudication/raw/second/0125.attempt0.json

results/adjudication/raw/second/0126.attempt0.json

results/adjudication/raw/second/0127.attempt0.json

results/adjudication/raw/second/0127.attempt1.json

results/adjudication/raw/second/0128.attempt0.json

results/adjudication/raw/second/0129.attempt0.json

results/adjudication/raw/second/0130.attempt0.json

results/adjudication/raw/second/0131.attempt0.json

results/adjudication/raw/second/0132.attempt0.json

results/adjudication/raw/second/0133.attempt0.json

results/adjudication/raw/second/0134.attempt0.json

results/adjudication/raw/second/0135.attempt0.json

results/adjudication/raw/second/0136.attempt0.json

results/adjudication/raw/second/0137.attempt0.json

results/adjudication/raw/second/0138.attempt0.json

results/adjudication/raw/second/0139.attempt0.json

results/adjudication/raw/second/0140.attempt0.json

results/adjudication/raw/second/0141.attempt0.json

results/adjudication/raw/second/0142.attempt0.json

results/adjudication/raw/second/0143.attempt0.json

results/adjudication/raw/second/0144.attempt0.json

results/adjudication/raw/second/0145.attempt0.json

results/adjudication/raw/second/0146.attempt0.json

results/adjudication/raw/second/0147.attempt0.json

results/adjudication/raw/second/0148.attempt0.json

results/adjudication/raw/second/0149.attempt0.json

results/adjudication/raw/second/0150.attempt0.json

results/adjudication/raw/second/0151.attempt0.json

results/adjudication/raw/second/0152.attempt0.json

results/adjudication/raw/second/0153.attempt0.json

results/adjudication/raw/second/0154.attempt0.json

results/adjudication/raw/second/0155.attempt0.json

results/adjudication/raw/second/0156.attempt0.json

results/adjudication/raw/second/0157.attempt0.json

results/adjudication/raw/second/0158.attempt0.json

results/adjudication/raw/second/0159.attempt0.json

results/adjudication/raw/second/0160.attempt0.json

results/adjudication/raw/second/0161.attempt0.json

results/adjudication/raw/second/0162.attempt0.json

results/adjudication/raw/second/0163.attempt0.json

results/adjudication/raw/second/0164.attempt0.json

results/adjudication/raw/second/0165.attempt0.json

results/adjudication/raw/second/0166.attempt0.json

results/adjudication/raw/second/0167.attempt0.json

results/adjudication/raw/second/0168.attempt0.json

results/adjudication/raw/second/0169.attempt0.json

results/adjudication/raw/second/0170.attempt0.json

results/adjudication/raw/second/0171.attempt0.json

results/adjudication/raw/second/0172.attempt0.json

results/adjudication/raw/second/0173.attempt0.json

results/adjudication/raw/second/0173.attempt1.json

results/adjudication/raw/second/0174.attempt0.json

results/adjudication/raw/second/0175.attempt0.json

results/adjudication/raw/second/0176.attempt0.json

results/adjudication/raw/second/0177.attempt0.json

results/adjudication/raw/second/0177.attempt1.json

results/adjudication/raw/second/0178.attempt0.json

results/adjudication/raw/second/0179.attempt0.json

results/adjudication/raw/second/0180.attempt0.json

results/adjudication/raw/second/0181.attempt0.json

results/adjudication/raw/second/0182.attempt0.json

results/adjudication/raw/second/0183.attempt0.json

results/adjudication/raw/second/0184.attempt0.json

results/adjudication/raw/second/0185.attempt0.json

results/adjudication/raw/second/0186.attempt0.json

results/adjudication/raw/second/0187.attempt0.json

results/adjudication/raw/second/0188.attempt0.json

results/adjudication/raw/second/0189.attempt0.json

results/adjudication/raw/second/0190.attempt0.json

results/adjudication/raw/second/0191.attempt0.json

results/adjudication/raw/second/0192.attempt0.json

results/adjudication/raw/second/0193.attempt0.json

results/adjudication/raw/second/0194.attempt0.json

results/adjudication/raw/second/0195.attempt0.json

results/adjudication/raw/second/0196.attempt0.json

results/adjudication/raw/second/0197.attempt0.json

results/adjudication/raw/second/0198.attempt0.json

results/adjudication/raw/second/0199.attempt0.json

results/adjudication/raw/second/0200.attempt0.json

results/adjudication/raw/second/0201.attempt0.json

results/adjudication/raw/second/0202.attempt0.json

results/adjudication/raw/second/0203.attempt0.json

results/adjudication/raw/second/0204.attempt0.json

results/adjudication/raw/second/0205.attempt0.json

results/adjudication/raw/second/0206.attempt0.json

results/adjudication/raw/second/0207.attempt0.json

results/adjudication/raw/second/0208.attempt0.json

results/adjudication/raw/second/0209.attempt0.json

results/adjudication/raw/second/0210.attempt0.json

results/adjudication/raw/second/0211.attempt0.json

results/adjudication/raw/second/0212.attempt0.json

results/adjudication/raw/second/0213.attempt0.json

results/adjudication/raw/second/0214.attempt0.json

results/adjudication/raw/second/0215.attempt0.json

results/adjudication/raw/second/0216.attempt0.json

results/adjudication/raw/second/0217.attempt0.json

results/adjudication/raw/second/0218.attempt0.json

results/adjudication/raw/second/0219.attempt0.json

results/adjudication/raw/second/0220.attempt0.json

results/adjudication/raw/second/0221.attempt0.json

results/adjudication/raw/second/0222.attempt0.json

results/adjudication/raw/second/0223.attempt0.json

results/adjudication/raw/second/0224.attempt0.json

results/adjudication/raw/second/0225.attempt0.json

results/adjudication/raw/second/0226.attempt0.json

results/adjudication/raw/second/0227.attempt0.json

results/adjudication/raw/second/0228.attempt0.json

results/adjudication/raw/second/0228.attempt1.json

results/adjudication/raw/second/0229.attempt0.json

results/adjudication/raw/second/0230.attempt0.json

results/adjudication/raw/second/0231.attempt0.json

results/adjudication/raw/second/0232.attempt0.json

results/adjudication/raw/second/0233.attempt0.json

results/adjudication/raw/second/0234.attempt0.json

results/adjudication/raw/second/0235.attempt0.json

results/adjudication/raw/second/0236.attempt0.json

results/adjudication/raw/second/0237.attempt0.json

results/adjudication/raw/second/0238.attempt0.json

results/adjudication/raw/second/0239.attempt0.json

results/adjudication/raw/second/0240.attempt0.json

results/adjudication/raw/second/0241.attempt0.json

results/adjudication/raw/second/0242.attempt0.json

results/adjudication/raw/second/0243.attempt0.json

results/adjudication/raw/second/0244.attempt0.json

results/adjudication/raw/second/0245.attempt0.json

results/adjudication/raw/second/0246.attempt0.json

results/adjudication/raw/second/0247.attempt0.json

results/adjudication/raw/second/0248.attempt0.json

results/adjudication/raw/second/0248.attempt1.json

results/adjudication/raw/second/0249.attempt0.json

results/adjudication/raw/second/0250.attempt0.json

results/adjudication/raw/second/0251.attempt0.json

results/adjudication/raw/second/0252.attempt0.json

results/adjudication/raw/second/0253.attempt0.json

results/adjudication/raw/second/0254.attempt0.json

results/adjudication/raw/second/0255.attempt0.json

results/adjudication/raw/second/0256.attempt0.json

results/adjudication/raw/second/0257.attempt0.json

results/adjudication/raw/second/0258.attempt0.json

results/adjudication/raw/second/0259.attempt0.json

results/adjudication/raw/second/0260.attempt0.json

results/adjudication/raw/second/0261.attempt0.json

results/adjudication/raw/second/0261.attempt1.json

results/adjudication/raw/second/0261.attempt2.json

results/adjudication/raw/second/0262.attempt0.json

results/adjudication/raw/second/0263.attempt0.json

results/adjudication/raw/second/0264.attempt0.json

results/adjudication/raw/second/0265.attempt0.json

results/adjudication/raw/second/0266.attempt0.json

results/adjudication/raw/second/0267.attempt0.json

results/adjudication/raw/second/0268.attempt0.json

results/adjudication/raw/second/0269.attempt0.json

results/adjudication/raw/second/0270.attempt0.json

results/adjudication/raw/second/0271.attempt0.json

results/adjudication/raw/tie_break/0000.attempt0.json

results/adjudication/raw/tie_break/0001.attempt0.json

results/adjudication/raw/tie_break/0002.attempt0.json

results/adjudication/raw/tie_break/0003.attempt0.json

results/adjudication/raw/tie_break/0004.attempt0.json

results/adjudication/raw/tie_break/0005.attempt0.json

results/adjudication/reader-first.jsonl

results/adjudication/reader-second.jsonl

results/adjudication/reader-tie_break.jsonl

results/adjudication/run-first.log

results/adjudication/run-second.log

results/adjudication/run-tie_break.log

results/adjudication/sample.jsonl

results/analysis-before-qwen-repair.json

results/analysis-grid-and-larger-pairs.json

results/analysis-grid.json

results/analysis-token-limit-rule.json

results/answer-tags.json

results/boxing-by-prompt.json

results/by-condition.json

results/common-cause.json

results/correctness-criterion.json

results/denominator-weight.json

results/dose-response-pooled.json

results/dose-response.json

results/external-agreement-direction.json

results/external-agreement.json

results/external-matched.json

results/external-reference.json

results/extraction-rate-settings.json

results/format-compliance.json

results/larger-pairs.json

results/llama-pilot-arc_c.json

results/llama-pilot.json

results/mechanism-by-prompt.json

results/mechanism-main-pairs.json

results/mechanism.json

results/omega-matched.json

results/omega-sensitivity.json

results/parser-departures.json

results/parser-reach-matrix-nonempty.json

results/parser-reach-matrix.json

results/published-bounds.json

results/published-vs-reread.json

results/qwen-repair-impact.json

results/release-loop.json

results/running-example-misses.json

results/running-example.json

results/share-checks.json

results/timings.json

results/truncation-check.json

results/with-letter-step/INDEX.json

results/with-letter-step/adjudication-result.json

results/with-letter-step/analysis-grid-and-larger-pairs.json

results/with-letter-step/analysis-grid.json

results/with-letter-step/by-condition.json

results/with-letter-step/common-cause.json

results/with-letter-step/denominator-weight.json

results/with-letter-step/dose-response-pooled.json

results/with-letter-step/dose-response.json

results/with-letter-step/external-matched.json

results/with-letter-step/external-reference.json

results/with-letter-step/extraction-rate-settings.json

results/with-letter-step/larger-pairs.json

results/with-letter-step/llama-pilot-arc_c.json

results/with-letter-step/llama-pilot.json

results/with-letter-step/parser-reach-matrix-nonempty.json

results/with-letter-step/parser-reach-matrix.json

results/with-letter-step/published-bounds.json

results/with-letter-step/published-vs-reread.json

results/with-letter-step/running-example-misses.json

results/with-letter-step/share-checks.json

runs/llama-pilot-arc_c/out/arc_c.base.jsonl

runs/llama-pilot-arc_c/out/arc_c.base.meta.json

runs/llama-pilot-arc_c/out/arc_c.rl.jsonl

runs/llama-pilot-arc_c/out/arc_c.rl.meta.json

runs/llama-pilot-mmlu/out/arc_c.base.jsonl

runs/llama-pilot-mmlu/out/arc_c.base.meta.json

runs/llama-pilot-mmlu/out/arc_c.rl.jsonl

runs/llama-pilot-mmlu/out/arc_c.rl.meta.json

runs/llama-pilot-mmlu/out/mmlu.base.jsonl

runs/llama-pilot-mmlu/out/mmlu.base.meta.json

runs/llama-pilot-mmlu/out/mmlu.rl.jsonl

runs/llama-pilot-mmlu/out/mmlu.rl.meta.json

runs/llama-pilot-mmlu/out/mmlu_pro.base.jsonl

runs/llama-pilot-mmlu/out/mmlu_pro.base.meta.json

runs/llama-pilot-mmlu/out/mmlu_pro.rl.jsonl

runs/llama-pilot-mmlu/out/mmlu_pro.rl.meta.json

(744 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.