analiza aiqbench
This research introduces GAC, a noise-aware adaptive controller that dynamically balances supervised fine-tuning and reinforcement learning signals during hybrid post-training. By estimating gradient noise variance online, the method achieves significant performance gains on math and code benchmarks while substantially reducing KL-drift compared to fixed mixing schedules. The approach demonstrates scalability across model sizes from 1.5B to 14B with minimal computational overhead.
GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
arXiv:2605.26184v2 Announce Type: replace-cross Abstract: Hybrid post-training combining supervised fine-tuning (SFT) and reinforcement learning (RL) is the standard paradigm for aligning large language models, yet fixed mixing schedules cannot adapt when the relative noise of the two signals evolves. We derive a noise-aware mixing weight mu* by minimizing an MSE upper bound on the mixed stochastic gradient, yielding a closed-form that balances gradient noise variance and SFT-RL disagreement. Building on the token-wise reweighting of CHORD, the practical Guided Adaptive Controller (GAC) adds EMA smoothing, a schedule prior, and capped updates around this estimator, with all statistics estimated online from existing training tensors. The noise-aware controller alone outperforms the best rule-based controller by +3.0 percentage points on AMC; the full system reaches +3.8 points over HPT across math, code, science, and logic benchmarks, while reducing KL-drift area by 28% and large absolute mixing-weight changes by more than 70%, at less than 1% overhead. Gains grow with model size from 1.5B to 14B. Resources: Qwen3.5-4B compact-backbone release: https://huggingface.co/YueLinHu/GAC-Qwen3.5-4B; reference implementation: https://github.com/deepnovacore/GAC; GAC project page: https://deepnovacore.github.io/GAC/
Abstract:Hybrid post-training combining supervised fine-tuning (SFT) and reinforcement learning (RL) is the standard paradigm for aligning large language models, yet fixed mixing schedules cannot adapt when the relative noise of the two signals evolves. We derive a noise-aware mixing weight mu* by minimizing an MSE upper bound on the mixed stochastic gradient, yielding a closed-form that balances gradient noise variance and SFT-RL disagreement. Building on the token-wise reweighting of CHORD, the practical Guided Adaptive Controller (GAC) adds EMA smoothing, a schedule prior, and capped updates around this estimator, with all statistics estimated online from existing training tensors. The noise-aware controller alone outperforms the best rule-based controller by +3.0 percentage points on AMC; the full system reaches +3.8 points over HPT across math, code, science, and logic benchmarks, while reducing KL-drift area by 28% and large absolute mixing-weight changes by more than 70%, at less than 1% overhead. Gains grow with model size from 1.5B to 14B. Resources: Qwen3.5-4B compact-backbone release: this https URL reference implementation: this https URL GAC project page: this https URL
17 pages, 3 figures, 22 tables. Camera-ready update adds links to the released Qwen3.5-4B compact-backbone checkpoint, reference implementation, and project page; experimental results and conclusions are unchanged
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.26184 [cs.LG]
(or arXiv:2605.26184v2 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 215 cuvinte
Subiecte conexe
News
Quantum Computing Report · 25 iul. 2026
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · 23 iul. 2026
News
Quantum Computing Report · 7 iun. 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · 21 iul. 2026