aiqbench analysis
This paper introduces ConSPO, a novel reinforcement learning algorithm that reformulates Group Relative Policy Optimization (GRPO) using contrastive sequence-level objectives to address limitations in score alignment and credit assignment. By optimizing length-normalized log-probabilities against verified positive and negative rollouts, the method aims to improve reasoning capabilities in large language models more effectively than current baselines. The work represents a significant theoretical and practical advancement in the post-training phase of AI models, specifically targeting the efficiency and accuracy of reward-driven learning processes.
Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
arXiv:2605.12969v4 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks. We first show that GRPO admits an equivalent discriminative reformulation, in which policy optimization maximizes the expected score gap between verified positive and negative rollouts. This reformulation reveals two objective-level limitations: likelihood-misaligned surrogate scores, in which clipped ratio-based scores are optimized rather than the sequence likelihoods that govern generation, and score-insensitive credit assignment, in which rollout-level credit does not reflect the current score gaps between positive and negative rollouts. To address these limitations, we propose ConSPO, a Contrastive Sequence-level Policy Optimization method that uses length-normalized sequence log-probabilities as rollout scores and contrasts verified positive rollouts against negative distractors within the same group. ConSPO optimizes a group-wise InfoNCE-style objective to adaptively strengthen updates for poorly separated positives and high-scoring negatives, together with a curriculum-scheduled margin that preserves separation pressure as training progresses. Experiments across diverse settings show that ConSPO outperforms strong baselines on challenging reasoning benchmarks.
Abstract:Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks. We first show that GRPO admits an equivalent discriminative reformulation, in which policy optimization maximizes the expected score gap between verified positive and negative rollouts. This reformulation reveals two objective-level limitations: likelihood-misaligned surrogate scores, in which clipped ratio-based scores are optimized rather than the sequence likelihoods that govern generation, and score-insensitive credit assignment, in which rollout-level credit does not reflect the current score gaps between positive and negative rollouts. To address these limitations, we propose ConSPO, a Contrastive Sequence-level Policy Optimization method that uses length-normalized sequence log-probabilities as rollout scores and contrasts verified positive rollouts against negative distractors within the same group. ConSPO optimizes a group-wise InfoNCE-style objective to adaptively strengthen updates for poorly separated positives and high-scoring negatives, together with a curriculum-scheduled margin that preserves separation pressure as training progresses. Experiments across diverse settings show that ConSPO outperforms strong baselines on challenging reasoning benchmarks.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.12969 [cs.LG]
(or arXiv:2605.12969v4 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.12969 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 262 words
Related coverage
When Partial Credit Opposes Completion
Snorkel AI Blog · Oct 15, 2026
ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
arXiv · Sep 30, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
EAGER: Enhancing Generative Event Extraction via Reinforcement Learning with Verifiable Rewards
arXiv · Sep 25, 2026