analiza aiqbench
This research identifies a critical flaw in Group Relative Policy Optimization (GRPO) where the advantage estimator inadvertently rewards guessing over reasoning, particularly in bounded-answer tasks. The authors introduce SIGNBALANCE, a composition-free magnitude method that restores zero-mean balance to prevent policies from converging toward spurious guess-like behaviors. By improving performance on math and search benchmarks without sacrificing open-answer capabilities, this work advances the reliability of reinforcement learning with verifiable rewards.
Spurious Advantage Hidden in GRPO
arXiv:2609.04063v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.
Abstract:Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.
Artificial Intelligence (cs.AI)
arXiv:2609.04063 [cs.AI]
(or arXiv:2609.04063v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.04063 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 258 cuvinte
Subiecte conexe
Custom reward functions for multi-turn reinforcement learning with Amazon Nova Forge
AWS · 14 aug. 2026
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
arXiv · 10 iun. 2026
Semifactual Credit-Augmented Policy Optimization
arXiv · 1 oct. 2026
Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning
arXiv · 25 aug. 2026