aiqbench analysis
Composing insight from the full article — this may take a moment.
Variance Reduction Based Experience Replay for Policy Optimization
arXiv:2602.05379v3 Announce Type: replace-cross Abstract: Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization. However, classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To address this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce the variance of policy gradient estimates. VRER is algorithm-agnostic and can be integrated with existing policy optimization methods, yielding the sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). To provide rigorous theoretical guarantees, we develop a novel analysis framework for experience replay that explicitly accounts for dependencies induced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and characterize a fundamental bias-variance trade-off: reusing older samples reduces gradient variance but may introduce greater estimation bias. Extensive experiments show that VRER consistently accelerates learning and outperforms state-of-the-art policy optimization algorithms
Abstract:Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization. However, classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To address this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce the variance of policy gradient estimates. VRER is algorithm-agnostic and can be integrated with existing policy optimization methods, yielding the sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). To provide rigorous theoretical guarantees, we develop a novel analysis framework for experience replay that explicitly accounts for dependencies induced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and characterize a fundamental bias-variance trade-off: reusing older samples reduces gradient variance but may introduce greater estimation bias. Extensive experiments show that VRER consistently accelerates learning and outperforms state-of-the-art policy optimization algorithms
46 pages, 6 figures. arXiv admin note: text overlap with arXiv:2208.12341
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2602.05379 [stat.ML]
(or arXiv:2602.05379v3 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2602.05379 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 262 words
Related coverage
Beyond Scaling: Self-Evolving LLM Agents for Hardware Kernel Optimization via an Experience-Driven Workflow and Experience Graph Memory
arXiv · Aug 27, 2026
Inverse Critical Experiment Design via Gradient Optimization and a Multigroup Attention-Based Neural Network Architecture
arXiv · Jun 5, 2026
Experience Augmented Policy Optimization for LLM Reasoning
arXiv · Jul 23, 2026
Trading Strategy Optimization via Textual Gradient
arXiv · Oct 6, 2026