aiqbench analysis
Bellman Policy Optimization (BPO) presents a critic-free reinforcement learning approach derived from Policy Mirror Descent, specifically tailored to improve the mathematical reasoning capabilities of large language models using verifiable rewards. By reformulating the optimization objective via Bellman equations, this method eliminates the need for intermediate state value estimation while maintaining theoretical equivalence to its predecessor. Experimental results on reasoning benchmarks confirm its effectiveness in enhancing model performance without relying on traditional critic networks.
Bellman Policy Optimization
arXiv:2609.15987v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.
Abstract:Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.
Machine Learning (cs.LG); Computation and Language (cs.CL); Optimization and Control (math.OC)
arXiv:2609.15987 [cs.LG]
(or arXiv:2609.15987v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.15987 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 205 words
Related coverage
Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
arXiv · Jun 9, 2026
Lifted Bellman Linear Programming for Offline Reinforcement Learning
arXiv · Sep 22, 2026
Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing
arXiv · Sep 1, 2026
Nonpreemptive Scheduling While Learning Context-Dependent Service Rates
arXiv · Sep 30, 2026