aiqbench analysis
This paper introduces RoVR-GSPO, a novel optimization framework designed to mitigate outlier sensitivity in group-relative policy optimization by decoupling reward and sequence-weight estimation channels. Through rigorous stability analysis and experiments on mathematical reasoning and tool-use tasks, the authors demonstrate superior robustness against reward contamination and token-level perturbations compared to existing methods. The work advances the reliability of reinforcement learning algorithms critical for high-stakes AI applications where data integrity is paramount.
Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation
arXiv:2609.36944v1 Announce Type: new Abstract: Group-relative policy optimization relies on reward-derived advantages and sequence-level likelihood weights, both of which can be sensitive to localized outliers. Extreme rewards can collapse the contrast among clean responses after group normalization, while token-level log-ratio perturbations can alter sequence weights and clipping decisions. We introduce RoVR-GSPO, a dual-channel robust optimizer that addresses these failure modes separately. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. We provide stability and efficiency analyses for both channels. Experiments on mathematical reasoning, long-context summarization, and tool-call annotation show consistent improvements over GSPO, while controlled perturbation studies demonstrate stronger robustness to reward contamination and token-ratio anomalies.
Abstract:Group-relative policy optimization relies on reward-derived advantages and sequence-level likelihood weights, both of which can be sensitive to localized outliers. Extreme rewards can collapse the contrast among clean responses after group normalization, while token-level log-ratio perturbations can alter sequence weights and clipping decisions. We introduce RoVR-GSPO, a dual-channel robust optimizer that addresses these failure modes separately. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. We provide stability and efficiency analyses for both channels. Experiments on mathematical reasoning, long-context summarization, and tool-call annotation show consistent improvements over GSPO, while controlled perturbation studies demonstrate stronger robustness to reward contamination and token-ratio anomalies.
Artificial Intelligence (cs.AI)
arXiv:2609.36944 [cs.AI]
(or arXiv:2609.36944v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.36944 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 210 words
