analiza aiqbench
This paper advances theoretical reinforcement learning by proving convergence guarantees for policy gradient methods in diffusion-based multi-armed bandit environments using stochastic differential equations. The authors establish a non-asymptotic regret bound of $O(\log T)$ and introduce a novel Lyapunov function that clarifies the analysis across both continuous-time SDE frameworks and discrete-time algorithms. These results provide rigorous mathematical foundations for applying policy gradient techniques to complex diffusion models, which is critical for improving stability in next-generation AI systems.
Convergence and Regret of the Policy Gradient for Multi-Armed Bandits in Diffusion Environment
arXiv:2607.29593v1 Announce Type: new Abstract: This paper studies the policy gradient update for a multi-arm bandit problem in diffusion environment that is described by a stochastic differential equation (SDE) under the continuous-time reinforcement learning framework by Wang et al. (2020), Jia and Zhou (2022b). With the logit parameterization for the stochastic policy, we show that it converges almost surely to the optimal arm under an arbitrary constant learning rate. Furthermore, we derive the non-asymptotic regret upper bound when the constant learning rate is below a time-invariant threshold; and the regret bound has order $O(\log T)$. We improve the analysis in Lattimore (2026a) for the same SDE by constructing a novel Lyapunov function and demonstrate the transparency of analyzing policy gradient using the tools in SDEs. In addition, the same Lyapunov function is also helpful in analyzing the discrete-time policy gradient algorithm.
Abstract:This paper studies the policy gradient update for a multi-arm bandit problem in diffusion environment that is described by a stochastic differential equation (SDE) under the continuous-time reinforcement learning framework by Wang et al. (2020), Jia and Zhou (2022b). With the logit parameterization for the stochastic policy, we show that it converges almost surely to the optimal arm under an arbitrary constant learning rate. Furthermore, we derive the non-asymptotic regret upper bound when the constant learning rate is below a time-invariant threshold; and the regret bound has order $O(\log T)$. We improve the analysis in Lattimore (2026a) for the same SDE by constructing a novel Lyapunov function and demonstrate the transparency of analyzing policy gradient using the tools in SDEs. In addition, the same Lyapunov function is also helpful in analyzing the discrete-time policy gradient algorithm.
Machine Learning (cs.LG); Optimization and Control (math.OC)
arXiv:2607.29593 [cs.LG]
(or arXiv:2607.29593v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.29593 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 234 cuvinte
Subiecte conexe
The Greedy Advantage in Finite-Horizon Bandits
arXiv · 3 aug. 2026
Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits
arXiv · 2 oct. 2026
Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
arXiv · 30 sept. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026