analiza aiqbench
This paper introduces PACS, a novel framework that stabilizes Reinforcement Learning with Verifiable Rewards (RLVR) for large language models by reformulating policy optimization as a supervised learning task. By treating rewards as predictable labels and leveraging implicit actor-critic coupling, the method achieves significant performance gains over existing baselines while recovering classical policy gradient updates through cross-entropy loss. This approach offers a promising pathway to address sparse reward signals and training instability in advanced LLM post-training pipelines for reasoning tasks like math and coding.
Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR
arXiv:2509.02522v3 Announce Type: replace Abstract: Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have empowered large language models (LLMs) to tackle challenging reasoning tasks such as mathematics and programming, however existing RLVR methods often suffer from sparse reward signals and unstable policy gradient updates inherent to RL-based approaches. To address the challenges, we propose PACS, a novel RLVR framework that achieves imPlicit Actor Critic coupling via a Supervised learning framework. By treating the outcome reward as a predictable label, we reformulate the RLVR problem into a supervised learning task over a score function parameterized by the policy model and optimized using cross-entropy loss. A detailed gradient analysis shows that this supervised formulation inherently recovers the classical policy gradient update while providing more stable and efficient training. Extensive experiments demonstrate that PACS significantly outperforms strong open-source models and RLVR baselines, yielding substantial average gains of +8.26% (4B) and +9.57% (8B) over base models offering a promising avenue for LLMs post-training with verifiable rewards. Our code and data are available as open source at https://github.com/ritzz-ai/PACS.
Abstract:Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have empowered large language models (LLMs) to tackle challenging reasoning tasks such as mathematics and programming, however existing RLVR methods often suffer from sparse reward signals and unstable policy gradient updates inherent to RL-based approaches. To address the challenges, we propose PACS, a novel RLVR framework that achieves imPlicit Actor Critic coupling via a Supervised learning framework. By treating the outcome reward as a predictable label, we reformulate the RLVR problem into a supervised learning task over a score function parameterized by the policy model and optimized using cross-entropy loss. A detailed gradient analysis shows that this supervised formulation inherently recovers the classical policy gradient update while providing more stable and efficient training. Extensive experiments demonstrate that PACS significantly outperforms strong open-source models and RLVR baselines, yielding substantial average gains of +8.26% (4B) and +9.57% (8B) over base models offering a promising avenue for LLMs post-training with verifiable rewards. Our code and data are available as open source at this https URL.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2509.02522 [cs.CL]
(or arXiv:2509.02522v3 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 186 cuvinte
Subiecte conexe
Anticipatory Risk-Guided Reinforcement Learning for Safe Flight Through Dynamic Clutter
arXiv · 28 iul. 2026
Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face · 11 mai 2026
From Tokens to Regions: CUDA-Sensitive Instruction Tuning for GPU Kernel Generation
arXiv · 16 iun. 2026
Extremely Sparse Supervision Incentivizes Reasoning Ability
arXiv · 7 sept. 2026