analiza aiqbench
This research proposes a novel post-training technique for Large Language Models that integrates Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization by averaging logits, thereby eliminating the need for KL regularization or a critic network. The method demonstrates improved accuracy on reasoning benchmarks like MATH and MMLU while preserving formatting capabilities from the reference policy.
Complementing reinforcement learning with SFT through logit averaging in the post training of LLMs
arXiv:2605.20555v2 Announce Type: replace-cross Abstract: We introduce a novel method that averages the logits of a frozen reference policy (e.g., SFT) and a trainable policy, and incorporate the method into Group Relative Policy Optimization (GRPO). In contrast to Reinforcement Learning with Verifiable Rewards (RLVR) methods, our proposal does not involve a Kullback Leibler (KL) regularization or critic; the trainable policy and the reference anchor are coupled through the logit averaging structure to leverage the reasoning expertise of the trainable policy while maintaining the formatting advantage of SFT. Our method is evaluated on MATH, cn-k12, and MMLU, and the results show a higher accuracy or at least comparable accuracy relative to the canonical KL-regularized GRPO.
Abstract:We introduce a novel method that averages the logits of a frozen reference policy (e.g., SFT) and a trainable policy, and incorporate the method into Group Relative Policy Optimization (GRPO). In contrast to Reinforcement Learning with Verifiable Rewards (RLVR) methods, our proposal does not involve a Kullback Leibler (KL) regularization or critic; the trainable policy and the reference anchor are coupled through the logit averaging structure to leverage the reasoning expertise of the trainable policy while maintaining the formatting advantage of SFT. Our method is evaluated on MATH, cn-k12, and MMLU, and the results show a higher accuracy or at least comparable accuracy relative to the canonical KL-regularized GRPO.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.20555 [cs.LG]
(or arXiv:2605.20555v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.20555 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 204 cuvinte
Subiecte conexe
When Partial Credit Opposes Completion
Snorkel AI Blog · 15 oct. 2026
Building multi-Region visualizations with Highcharts in Amazon Quick
AWS · 23 iul. 2026
No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation
arXiv · 1 oct. 2026
Granite 4.1 LLMs: How They’re Built
Hugging Face · 29 apr. 2026