analiza aiqbench
This paper is squarely about AI: it analyzes estimation bias in group-relative advantage methods like GRPO used to post-train large language models with verifier rewards. The core finding is that mean-centered advantages systematically misweight correct answers on hard prompts and incorrect answers on easy prompts, and the proposed HA-DW reweighting is evaluated on five math reasoning benchmarks. For readers, the claims are useful but still early: the gains are reported on math tasks with binary rewards, so generalization to other RLVR domains remains to be shown.
Your Group-Relative Advantage Is Biased
arXiv:2601.08521v3 Announce Type: replace Abstract: Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we characterize conditional estimation bias in group-relative advantages among non-degenerate groups. For mean-centered advantages under binary rewards, our analysis shows negative conditional mean estimation errors for hard prompts and positive errors for easy prompts. At the response level, correct answers to hard prompts are underweighted, while incorrect answers to easy prompts receive insufficient penalties. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.
Abstract:Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we characterize conditional estimation bias in group-relative advantages among non-degenerate groups. For mean-centered advantages under binary rewards, our analysis shows negative conditional mean estimation errors for hard prompts and positive errors for easy prompts. At the response level, correct answers to hard prompts are underweighted, while incorrect answers to easy prompts receive insufficient penalties. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.
Machine Learning (cs.LG)
arXiv:2601.08521 [cs.LG]
(or arXiv:2601.08521v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2601.08521 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 267 cuvinte