RCCM: Diagnosing Gains and Harms in Reasoning-Prompt Transfer

Abstract:A reasoning prompt can improve one deployment and degrade another. RCCM (Reasoning-Control Capability Mismatch) examines these reversals by pairing answers to the same questions with records of how generation ends. Across five open-weight deployments and nine strategies, aggregate accuracy hides substantial question-level gains and harms; a common-budget MATH500 evaluation examines transfer on a separate sample. On 60 additional questions, concise reasoning improves Phi's automatic answer matching at a 2,048-token cap across three collections, but sharply reduces Qwen's. Author-reviewed final-answer commitments give a smaller, statistically unresolved Phi gain. Reserving an answer stage also improves matching, with additional input and call costs. These results show why a prompt benefit must be interpreted alongside its execution and evaluation: matching a reference answer need not mean delivering it, and an apparent compression benefit may admit another explanation. RCCM records this evidence for reproducible transfer comparisons.

21 pages, 2 figures. Includes a size-limited reproducibility subset as ancillary files

Computation and Language (cs.CL)

arXiv:2610.19872 [cs.CL]

(or arXiv:2610.19872v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2610.19872 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

ARXIV_SUBSET_MANIFEST.json

ARXIV_SUBSET_README.md

RCCM_ARTIFACT/ASSET_SOURCES.md

RCCM_ARTIFACT/CITATION_AUDIT_36.json

RCCM_ARTIFACT/CURRENT_REFERENCE_AND_KIT_VERIFICATION.json

RCCM_ARTIFACT/FULL_ARTIFACT_README_ARCHIVED.md

RCCM_ARTIFACT/OFFLINE_ENVIRONMENT.json

RCCM_ARTIFACT/PUBLIC_MANIFEST.json

RCCM_ARTIFACT/PUBLIC_MANIFEST_R268.json

RCCM_ARTIFACT/PUBLIC_MANIFEST_R270.json

RCCM_ARTIFACT/PUBLIC_MANIFEST_R274.json

RCCM_ARTIFACT/PUBLIC_MANIFEST_R275.json

RCCM_ARTIFACT/PUBLIC_MANIFEST_R279.json

RCCM_ARTIFACT/README.md

RCCM_ARTIFACT/README_DATA.md

RCCM_ARTIFACT/README_SOFTWARE.md

RCCM_ARTIFACT/REPLAY_AND_COLLECTION.md

RCCM_ARTIFACT/SELECTION_AND_STATUS.md

RCCM_ARTIFACT/analysis/AUXILIARY_SCORE_AUDIT.json

RCCM_ARTIFACT/analysis/CITATION_FRESH_CHECK.json

RCCM_ARTIFACT/analysis/REPLAY_REPORT.json

RCCM_ARTIFACT/analysis/REVISION_VERIFICATION.json

RCCM_ARTIFACT/analysis/SEMANTIC_AUDIT.json

RCCM_ARTIFACT/analysis/candidate_paired_by_deployment.csv

RCCM_ARTIFACT/analysis/candidate_replay.csv

RCCM_ARTIFACT/analysis/candidate_summary.csv

RCCM_ARTIFACT/analysis/deployment_declarations.csv

RCCM_ARTIFACT/analysis/main_all_pairs.csv

RCCM_ARTIFACT/analysis/main_matrix.csv

RCCM_ARTIFACT/analysis/main_rank_correlations.csv

RCCM_ARTIFACT/analysis/math500_all_pairs.csv

RCCM_ARTIFACT/analysis/math500_summary.csv

RCCM_ARTIFACT/analysis/math500_top_bands.csv

RCCM_ARTIFACT/analysis/paired_effect_intervals.csv

RCCM_ARTIFACT/analysis/phi_joint.csv

RCCM_ARTIFACT/analysis/probe_conditions.csv

RCCM_ARTIFACT/analysis/probe_pairs.csv

RCCM_ARTIFACT/analysis/reextraction_disagreements.jsonl

RCCM_ARTIFACT/analysis/repair_input_score_audit.csv

RCCM_ARTIFACT/analysis/rescue_harm.csv

RCCM_ARTIFACT/analysis/sampling_accuracy_replayed.csv

RCCM_ARTIFACT/analysis/sampling_event_audit.csv

RCCM_ARTIFACT/analysis/sampling_vote_replay.csv

RCCM_ARTIFACT/analysis/split_profiles.json

RCCM_ARTIFACT/analysis/template_bindings.csv

RCCM_ARTIFACT/analysis/threshold_sensitivity.csv

RCCM_ARTIFACT/anonymous_data_MANIFEST.json

RCCM_ARTIFACT/anonymous_software_MANIFEST.json

RCCM_ARTIFACT/answer_annotations/ANNOTATION_ANALYSIS.json

RCCM_ARTIFACT/answer_annotations/INTAKE_PROVENANCE.json

RCCM_ARTIFACT/answer_annotations/SHA256_INPUTS.json

RCCM_ARTIFACT/answer_annotations/joined_annotations.csv

RCCM_ARTIFACT/answer_annotations/original_review_kit/KIT_STATUS.md

RCCM_ARTIFACT/answer_annotations/original_review_kit/README.md

RCCM_ARTIFACT/answer_annotations/original_review_kit/___________________________CN.md

RCCM_ARTIFACT/answer_annotations/original_review_kit/blinded_responses.html

RCCM_ARTIFACT/answer_annotations/original_review_kit/reviewer_1_blank.csv

RCCM_ARTIFACT/answer_annotations/original_review_kit/reviewer_2_blank.csv

RCCM_ARTIFACT/answer_annotations/received_annotations.csv

RCCM_ARTIFACT/answer_phase/FINAL_ANALYSIS.json

RCCM_ARTIFACT/answer_phase/FROZEN_PROTOCOL.json

RCCM_ARTIFACT/answer_phase/INDEPENDENT_ACCEPTANCE.json

RCCM_ARTIFACT/answer_phase/INDEPENDENT_STATISTICS.json

RCCM_ARTIFACT/answer_phase/PREFLIGHT.json

RCCM_ARTIFACT/answer_phase/PROTOCOL_CN.md

RCCM_ARTIFACT/answer_phase/RUNTIME_SNAPSHOT.json

RCCM_ARTIFACT/answer_phase/TECHNICAL_GATE.json

RCCM_ARTIFACT/answer_phase/analyze.py

RCCM_ARTIFACT/answer_phase/attempts.jsonl

RCCM_ARTIFACT/answer_phase/calls.jsonl

RCCM_ARTIFACT/answer_phase/common.py

RCCM_ARTIFACT/answer_phase/events.jsonl

RCCM_ARTIFACT/answer_phase/outcomes.jsonl

RCCM_ARTIFACT/answer_phase/requests.jsonl

RCCM_ARTIFACT/answer_phase/runner.py

RCCM_ARTIFACT/answer_phase/verify_completed.py

RCCM_ARTIFACT/confirmation/AMENDED_ANALYSIS.json

RCCM_ARTIFACT/confirmation/FROZEN_CLOSURE.json

RCCM_ARTIFACT/confirmation/FROZEN_CLOSURE_V002.json

RCCM_ARTIFACT/confirmation/FROZEN_CONFIG.json

RCCM_ARTIFACT/confirmation/INDEPENDENT_AUDIT.json

RCCM_ARTIFACT/confirmation/METADATA_REDACTION.json

RCCM_ARTIFACT/confirmation/RUNTIME_MANIFEST_ANON.json

RCCM_ARTIFACT/confirmation/answer_scope_sensitivity.json

RCCM_ARTIFACT/confirmation/freshness_audit.json

RCCM_ARTIFACT/confirmation/items_frozen.jsonl

RCCM_ARTIFACT/confirmation/outcomes.jsonl

RCCM_ARTIFACT/confirmation/paired_effects_audited.csv

RCCM_ARTIFACT/confirmation/requests_frozen.jsonl

RCCM_ARTIFACT/confirmation/response_structure_features.jsonl

RCCM_ARTIFACT/confirmation/templates_frozen.json

RCCM_ARTIFACT/consolidation/ANALYSIS_MANIFEST.json

RCCM_ARTIFACT/consolidation/ANALYSIS_PROTOCOL.md

RCCM_ARTIFACT/consolidation/EXPECTED_SUMMARIES.json

RCCM_ARTIFACT/data/candidate_archive/boundary_replay.json

RCCM_ARTIFACT/data/candidate_archive/candidate_aggregate_comparison.csv

RCCM_ARTIFACT/data/candidate_archive/candidate_cases_exact.csv

RCCM_ARTIFACT/data/main_panel_scores.csv

RCCM_ARTIFACT/data/probes/conditions_recomputed.csv

RCCM_ARTIFACT/data/probes/contrasts_recomputed.csv

RCCM_ARTIFACT/data/probes/gemma_verification.jsonl

RCCM_ARTIFACT/data/probes/phi_length.jsonl

RCCM_ARTIFACT/data/probes/qwen_structure.jsonl

RCCM_ARTIFACT/data/repair/repair_breakdown_expanded_gemma.jsonl

RCCM_ARTIFACT/data/repair/repair_breakdown_expanded_llama.jsonl

RCCM_ARTIFACT/data/repair/repair_breakdown_expanded_phi_truncated.jsonl

RCCM_ARTIFACT/data/repair/repair_breakdown_expanded_qwen.jsonl

RCCM_ARTIFACT/data/sampling_config_gemma.json

RCCM_ARTIFACT/data/sampling_config_llama.json

RCCM_ARTIFACT/data/sampling_config_phi.json

RCCM_ARTIFACT/data/sampling_config_qwen.json

RCCM_ARTIFACT/data/templates.json

RCCM_ARTIFACT/figures/fig1_rescue_harm.pdf

RCCM_ARTIFACT/figures/fig1_rescue_harm.png

RCCM_ARTIFACT/figures/fig1_rescue_harm.svg

RCCM_ARTIFACT/figures/fig2_phi_stops.pdf

RCCM_ARTIFACT/figures/fig2_phi_stops.png

RCCM_ARTIFACT/figures/fig2_phi_stops.svg

RCCM_ARTIFACT/historical_edits/factorial_marginals.csv

RCCM_ARTIFACT/official_sources/gsm8k_license.txt

RCCM_ARTIFACT/official_sources/math_license.txt

RCCM_ARTIFACT/provenance/repair_phi_runner_original.py

RCCM_ARTIFACT/provenance/repair_runner_original.py

RCCM_ARTIFACT/provenance/run_checkpoint.py

RCCM_ARTIFACT/provenance/run_new_methods.py

RCCM_ARTIFACT/provenance/run_repair.py

RCCM_ARTIFACT/provenance/sampling_analyzer_original.py

RCCM_ARTIFACT/references.bib

RCCM_ARTIFACT/references_historical.bib

RCCM_ARTIFACT/repeatability/COMPLETION_SNAPSHOT.json

RCCM_ARTIFACT/repeatability/FINAL_ANALYSIS.json

RCCM_ARTIFACT/repeatability/FROZEN_PROTOCOL.json

RCCM_ARTIFACT/repeatability/INDEPENDENT_ACCEPTANCE.json

RCCM_ARTIFACT/repeatability/RUNTIME_SNAPSHOT.json

RCCM_ARTIFACT/repeatability/analyze.py

RCCM_ARTIFACT/repeatability/attempts.jsonl

RCCM_ARTIFACT/repeatability/events.jsonl

RCCM_ARTIFACT/repeatability/outcomes.jsonl

RCCM_ARTIFACT/repeatability/requests.jsonl

RCCM_ARTIFACT/repeatability/runner.py

RCCM_ARTIFACT/requirements-lock.txt

RCCM_ARTIFACT/requirements.txt

RCCM_ARTIFACT/retrospective/expected_auc.csv

RCCM_ARTIFACT/retrospective/falcon_contrasts.csv

RCCM_ARTIFACT/retrospective/long_format_data.csv

RCCM_ARTIFACT/scripts/check_environment.py

RCCM_ARTIFACT/scripts/consolidated_analysis.py

RCCM_ARTIFACT/scripts/exact_rules.py

RCCM_ARTIFACT/scripts/plot_figures.py

RCCM_ARTIFACT/scripts/replay_answer_annotations.py

RCCM_ARTIFACT/scripts/replay_answer_phase.py

RCCM_ARTIFACT/scripts/replay_auxiliary_scores.py

RCCM_ARTIFACT/scripts/replay_confirmation.py

RCCM_ARTIFACT/scripts/replay_consolidation.py

RCCM_ARTIFACT/scripts/replay_followup_diagnostics.py

RCCM_ARTIFACT/scripts/replay_historical_edits.py

RCCM_ARTIFACT/scripts/replay_repeatability.py

RCCM_ARTIFACT/scripts/replay_retrospective_fit.py

RCCM_ARTIFACT/scripts/replay_selection.py

RCCM_ARTIFACT/scripts/reproduce_all.py

RCCM_ARTIFACT/scripts/run_all.py

RCCM_ARTIFACT/scripts/scorers/legacy/aaai_eval/math_match.py

RCCM_ARTIFACT/scripts/scorers/legacy/aaai_eval/parsers.py

RCCM_ARTIFACT/scripts/scorers/v005/aaai_eval/math_match.py

RCCM_ARTIFACT/scripts/scorers/v005/aaai_eval/parsers.py

RCCM_ARTIFACT/scripts/semantic_audit.py

RCCM_ARTIFACT/scripts/verify_answer_phase_analysis.py

RCCM_ARTIFACT/selection_sources/SELECTION_PROVENANCE.json

RCCM_ARTIFACT/selection_sources/gsm8k_test_source.jsonl

RCCM_ARTIFACT/selection_sources/math_test_algebra.parquet

RCCM_ARTIFACT/selection_sources/math_test_counting_and_probability.parquet

RCCM_ARTIFACT/selection_sources/math_test_geometry.parquet

RCCM_ARTIFACT/selection_sources/math_test_intermediate_algebra.parquet

RCCM_ARTIFACT/selection_sources/math_test_number_theory.parquet

RCCM_ARTIFACT/selection_sources/math_test_prealgebra.parquet

RCCM_ARTIFACT/selection_sources/math_test_precalculus.parquet

(171 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.