aiqbench analysis
This paper presents ToPPO, a novel off-policy extension to Proximal Policy Optimization (PPO) that theoretically justifies and computationally optimizes the use of disparate data sources for reinforcement learning. By establishing a new policy improvement lower bound, the method ensures monotonic performance gains while addressing PPO's traditional limitations in leveraging existing datasets. The approach offers significant potential for improving sample efficiency in complex RL tasks without compromising training stability.
Transductive Off-policy Proximal Policy Optimization
arXiv:2406.03894v2 Announce Type: replace Abstract: Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due to its inherent on-policy nature, its proficiency in harnessing data from disparate policies is constrained. This paper introduces a novel off-policy extension to the original PPO method, christened Transductive Off-policy PPO (ToPPO). Herein, we provide theoretical justification for incorporating off-policy data in PPO training and prudent guidelines for its safe application. Our contribution includes a novel formulation of the policy improvement lower bound for prospective policies derived from off-policy data, accompanied by a computationally efficient mechanism to optimize this bound, underpinned by assurances of monotonic improvement. Comprehensive experimental results across six representative tasks underscore ToPPO's promising performance.
Abstract:Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due to its inherent on-policy nature, its proficiency in harnessing data from disparate policies is constrained. This paper introduces a novel off-policy extension to the original PPO method, christened Transductive Off-policy PPO (ToPPO). Herein, we provide theoretical justification for incorporating off-policy data in PPO training and prudent guidelines for its safe application. Our contribution includes a novel formulation of the policy improvement lower bound for prospective policies derived from off-policy data, accompanied by a computationally efficient mechanism to optimize this bound, underpinned by assurances of monotonic improvement. Comprehensive experimental results across six representative tasks underscore ToPPO's promising performance.
Machine Learning (cs.LG)
arXiv:2406.03894 [cs.LG]
(or arXiv:2406.03894v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2406.03894 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 208 words
Related coverage
Variational Proximal Policy Optimization
arXiv · Jun 9, 2026
Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models
arXiv · Jun 10, 2026
Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark
AMD · Aug 18, 2026
BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning
arXiv · Sep 1, 2026