Small Experiments, Cheaper Decisions: A Case Study in Staged Promotion for Micro-Pretraining

Abstract:Short pretraining runs can reduce experimental cost, but they can also over-promote configurations that only look strong at tiny budgets. We study an auditable staged-promotion protocol for a fixed micro-pretraining runner on two heterogeneous host blocks: Windows A100 and Linux L40S. Starting from twelve prior-screened configurations, we use staged budgets of 2 minutes, 5 minutes, 10 minutes, 60 minutes, and 12 hours, with frozen promotion rules before expensive continuations. The early screens are intentionally treated as unstable: the 5- and 10-minute rankings are host-sensitive, and the eventual 12-hour top-ranked condition is not the mean-best condition at the replicated 10-minute gate. Because seed ranges differ across stages, these changes are operational promotion evidence, not within-seed curves. A replicated 60-minute gate keeps the Staged Factorial Screening bridge reference in the promoted set, where it ranks first in all four 60-minute host-seed cells. In the final 12-hour confirmation package, the bridge condition ranks first in all four host-seed cells across two seeds; the greedy comparator does not meet the frozen 0.010 val_bpb near-equivalence rule; and the cheaper d8/ar48 (depth-8, aspect-48) sentinel does not meet the frozen 0.020 mean-gap rule. The executed 12-hour branch spends 144 GPU-hours, and the full staged protocol records 169.2 training GPU-hours including screening stages. Continuing all four 60-minute candidates would spend 192 GPU-hours, while continuing all nine replicated 10-minute candidates would spend 432 GPU-hours. The latter numbers are accounting counterfactuals for unrun continuations, not evidence that skipped candidates could not have overtaken the reference. The result is a bounded cost-allocation finding, not a claim of global optimality, capacity-normalized superiority, or superiority over adaptive hyperparameter optimization methods.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2606.11387 [cs.CL]

(or arXiv:2606.11387v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.11387 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

analysis/p06_budget_counterfactual_2026-05-02.csv

analysis/p06_budget_counterfactual_2026-05-02.json

analysis/p06_promotion_survival_2026-05-02.csv

analysis/p06_promotion_survival_2026-05-02.json

analysis/p06_stage0_smoke_summary_2026-05-02.csv

analysis/p06_stage0_smoke_summary_2026-05-02.json

analysis/p06_threshold_sensitivity_2026-05-02.csv

analysis/p06_threshold_sensitivity_2026-05-02.json

analysis/stage1_10min_seed42_seed43_top9_promotion_decision.json

analysis/stage1_10min_seed42_seed43_top9_promotion_summary.csv

analysis/stage1_5min_host_comparison.csv

analysis/stage1_5min_host_comparison.json

analysis/stage2_60min_seed44_seed45_combined_decision.csv

analysis/stage2_60min_seed44_seed45_combined_decision.json

analysis/stage3_12h_seed46_seed47_combined_cells.csv

analysis/stage3_12h_seed46_seed47_combined_decision.json

analysis/stage3_12h_seed46_seed47_combined_summary.csv

control/README.txt

control/analyze_p06_offline_readiness_checks.py

control/analyze_stage3_12h_seed46_seed47.py

control/analyze_stage_summary.py

control/apply_stage1_promotion_gate.py

control/generate_paper06_figures.py

control/materialize_p06_runs.py

control/parse_train_summary.py

control/summarize_p06_runs.py

control/verify_checkpoint.py

figure_manifest.json

matrices/paper06_starter_matrix_2026-04-27.csv

preanalysis/EXPERIMENT_DESIGN_DECISION_2026-04-28.txt

preanalysis/PREANALYSIS_STAGE1_SEED43_TO_60MIN_2026-04-28.txt

preanalysis/PREANALYSIS_STAGE2_60MIN_SEED44_2026-04-28.txt

preanalysis/PREANALYSIS_STAGE3B_12H_SEED47_2026-04-30.txt

preanalysis/PREANALYSIS_STAGE3_12H_SEED46_2026-04-29.txt

preanalysis/STAGE1_5MIN_ANALYSIS_2026-04-27.txt

preanalysis/STAGE2_60MIN_SEED44_DECISION_2026-04-28.txt

preanalysis/STAGE2_60MIN_SEED45_DECISION_2026-04-29.txt

preanalysis/STAGE3B_12H_SEED47_DECISION_2026-05-02.txt

preanalysis/STAGE3_12H_SEED46_DECISION_2026-04-30.txt

remote-results/linux_stage0_summary.csv

remote-results/linux_stage1_10min_seed43_top9_summary.csv

remote-results/linux_stage1_10min_summary.csv

remote-results/linux_stage1_5min_summary.csv

remote-results/linux_stage2_60min_seed44_gate4_summary.csv

remote-results/linux_stage2_60min_seed45_gate4_summary.csv

remote-results/linux_stage3_12h_seed46_anchor3_summary.csv

remote-results/linux_stage3b_12h_seed47_anchor3_summary.csv

remote-results/windows_stage0_summary.csv

remote-results/windows_stage1_10min_seed43_top9_summary.csv

remote-results/windows_stage1_10min_summary.csv

remote-results/windows_stage1_5min_summary.csv

remote-results/windows_stage2_60min_seed44_gate4_summary.csv

remote-results/windows_stage2_60min_seed45_gate4_summary.csv

remote-results/windows_stage3_12h_seed46_anchor3_summary.csv

remote-results/windows_stage3b_12h_seed47_anchor3_summary.csv

source/runner__train_p06_instrumented.py

source/source__autoresearch-seed__prepare.py

source/source__autoresearch-seed__pyproject.toml

source/source__autoresearch-seed__train.py

table_manifest.json

(58 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.