analiza aiqbench
This paper is squarely about AI: a training method for large language models that adds an offline "golden" reasoning trace as a contrastive signal when all sampled responses fail. The core idea is a practical fix for the zero-reward dead zone in group-based RLVR, and the reported gains are on 10 math benchmarks at two model scales, so readers should check how well the results transfer beyond math and how much the golden-trajectory data costs to produce. This is a research-level result, not a product or deployment claim.
Group Contrastive Policy Optimization: Learning Beyond the Self-Exploration Limit with Golden Answers
arXiv:2510.07790v2 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving the reasoning abilities of large language models, yet group-based methods lose their learning signal when every sampled response receives zero reward. We propose Group Contrastive Policy Optimization (GCPO), a hybrid objective that preserves standard on-policy optimization for informative rollout groups and activates a proximal contrastive objective only when exploration completely fails. The auxiliary objective pairs an offline golden reasoning trajectory with the current policy's failed responses, thereby providing a reliable positive reference and prompt-specific hard negatives without treating the golden trajectory as an on-policy rollout. Its old-policy token probabilities are evaluated through teacher forcing and used solely to constrain the update, rather than as an importance-sampling correction for the teacher distribution. A first-order analysis shows that the resulting update increases a length-normalized likelihood margin between reliable reasoning and observed failures. Experiments across two model scales and 10 core mathematical reasoning benchmarks demonstrate that GCPO improves training efficiency and final reasoning performance over competitive supervised and RLVR baselines.
Abstract:Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving the reasoning abilities of large language models, yet group-based methods lose their learning signal when every sampled response receives zero reward. We propose Group Contrastive Policy Optimization (GCPO), a hybrid objective that preserves standard on-policy optimization for informative rollout groups and activates a proximal contrastive objective only when exploration completely fails. The auxiliary objective pairs an offline golden reasoning trajectory with the current policy's failed responses, thereby providing a reliable positive reference and prompt-specific hard negatives without treating the golden trajectory as an on-policy rollout. Its old-policy token probabilities are evaluated through teacher forcing and used solely to constrain the update, rather than as an importance-sampling correction for the teacher distribution. A first-order analysis shows that the resulting update increases a length-normalized likelihood margin between reliable reasoning and observed failures. Experiments across two model scales and 10 core mathematical reasoning benchmarks demonstrate that GCPO improves training efficiency and final reasoning performance over competitive supervised and RLVR baselines.
Artificial Intelligence (cs.AI)
arXiv:2510.07790 [cs.AI]
(or arXiv:2510.07790v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2510.07790 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 262 cuvinte