analiza aiqbench
This research proposes Proximal Entropy Policy Optimization (PEPO), a novel method that addresses the limitations of global entropy measures in Reinforcement Learning from Verifiable Rewards (RLVR). By introducing a local context-aware metric, PEPO effectively weights per-token advantages to improve mathematical reasoning capabilities across various open-source models. The study demonstrates that this formulation generalizes to existing algorithms and succeeds in single-stream RL scenarios where traditional approaches fail.
Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization
arXiv:2609.39402v1 Announce Type: new Abstract: Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.
Abstract:Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.
21 pages, 4 figures. Accepted at NeurIPS 2026
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2609.39402 [cs.AI]
(or arXiv:2609.39402v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.39402 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 240 cuvinte
Subiecte conexe
Reinforcement Learning from Rich Feedback with Distributional DAgger
arXiv · 8 iun. 2026
Reinforcement Learning from Rich Feedback with Distributional DAgger
arXiv · 5 iun. 2026
HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments
arXiv · 16 sept. 2026
Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning
arXiv · 30 sept. 2026