GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

Abstract:Hybrid post-training combining supervised fine-tuning (SFT) and reinforcement learning (RL) is the standard paradigm for aligning large language models, yet fixed mixing schedules cannot adapt when the relative noise of the two signals evolves. We derive a noise-aware mixing weight mu* by minimizing an MSE upper bound on the mixed stochastic gradient, yielding a closed-form that balances gradient noise variance and SFT-RL disagreement. Building on the token-wise reweighting of CHORD, the practical Guided Adaptive Controller (GAC) adds EMA smoothing, a schedule prior, and capped updates around this estimator, with all statistics estimated online from existing training tensors. The noise-aware controller alone outperforms the best rule-based controller by +3.0 percentage points on AMC; the full system reaches +3.8 points over HPT across math, code, science, and logic benchmarks, while reducing KL-drift area by 28% and large absolute mixing-weight changes by more than 70%, at less than 1% overhead. Gains grow with model size from 1.5B to 14B. Resources: Qwen3.5-4B compact-backbone release: this https URL reference implementation: this https URL GAC project page: this https URL

17 pages, 3 figures, 22 tables. Camera-ready update adds links to the released Qwen3.5-4B compact-backbone checkpoint, reference implementation, and project page; experimental results and conclusions are unchanged

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2605.26184 [cs.LG]

(or arXiv:2605.26184v2 [cs.LG] for this version)