analiza aiqbench
RASFT introduces a policy-aware fine-tuning framework that dynamically adjusts expert supervision based on real-time problem solvability, effectively balancing rigid imitation with the model's own reasoning capabilities. By incorporating verified on-policy rollouts and constraining policy drift, the method outperforms standard SFT and reinforcement learning approaches across mathematical and code reasoning benchmarks. This approach addresses the critical limitation of overfitting to surface forms in single-trajectory imitation, offering a more robust adaptation strategy for complex reasoning tasks.
RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning
arXiv:2606.07006v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) is a prevailing method for adapting large language models to reasoning tasks by imitating offline expert demonstrations, often treating a single expert trajectory as the target behavior. However, reasoning is not simple path imitation: rigidly following one demonstrated solution may overfit to surface forms and suppress the model's own reasoning distribution. We propose Rollout-Adaptive Supervised Fine-Tuning (RASFT), a policy-aware SFT framework that calibrates expert supervision according to problem-level solvability estimated from verified on-policy rollouts. For each problem, RASFT strengthens expert guidance when the current policy struggles, while relaxing rigid imitation and incorporating correct self-generated trajectories when the model already exhibits reliable reasoning behavior. To preserve useful reasoning priors, RASFT further introduces a clipped inverse ratio between the frozen reference model and the current policy to constrain excessive policy drift. Experiments across multiple models on six mathematical reasoning benchmarks and two code reasoning benchmarks show that RASFT achieves better overall performance than SFT, SFT variants, and representative RL methods. The code is available at https://github.com/zjd1sq/RASFT.
Abstract:Supervised fine-tuning (SFT) is a prevailing method for adapting large language models to reasoning tasks by imitating offline expert demonstrations, often treating a single expert trajectory as the target behavior. However, reasoning is not simple path imitation: rigidly following one demonstrated solution may overfit to surface forms and suppress the model's own reasoning distribution. We propose Rollout-Adaptive Supervised Fine-Tuning (RASFT), a policy-aware SFT framework that calibrates expert supervision according to problem-level solvability estimated from verified on-policy rollouts. For each problem, RASFT strengthens expert guidance when the current policy struggles, while relaxing rigid imitation and incorporating correct self-generated trajectories when the model already exhibits reliable reasoning behavior. To preserve useful reasoning priors, RASFT further introduces a clipped inverse ratio between the frozen reference model and the current policy to constrain excessive policy drift. Experiments across multiple models on six mathematical reasoning benchmarks and two code reasoning benchmarks show that RASFT achieves better overall performance than SFT, SFT variants, and representative RL methods. The code is available at this https URL.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2606.07006 [cs.LG]
(or arXiv:2606.07006v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.07006 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 267 cuvinte
Subiecte conexe
Improving CoT Monitorability of Evaluation Awareness via Verbalization Training
LessWrong · 2 oct. 2026
Welcome Beam: a 501B open model you can own, customize, and scale
Hugging Face · 16 oct. 2026
Fine-tune NVIDIA Nemotron 3 models with Amazon SageMaker AI serverless model customization
AWS · 10 iul. 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · 17 sept. 2026