aiqbench analysis
This paper advances reinforcement learning theory by framing policy optimization as a geometric problem within the Wasserstein space of action probabilities. By leveraging Otto's calculus, the authors establish a rigorous Riemannian structure for policies and provide a formal second-order analysis via gradient and Hessian computations. The approach bridges optimal transport theory with deep learning, offering new mathematical tools for understanding the landscape of policy optimization in both low-dimensional settings and high-dimensional neural network parameterizations.
Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
arXiv:2604.14765v2 Announce Type: replace Abstract: We present a geometric framework for Reinforcement Learning (RL) that views policies as maps into the Wasserstein space of action probabilities. First, we define a Riemannian structure induced by stationary distributions, proving its existence in a general context. We then define the tangent space of policies and characterize the geodesics, specifically addressing the measurability of vector fields mapped from the state space to the tangent space of probability measures over the action space. Next, we formulate a general RL optimization problem and construct a gradient flow using Otto's calculus. We compute the gradient and the Hessian of the energy, providing a formal second-order analysis. Finally, we illustrate the method with numerical examples for low-dimensional problems, computing the gradient directly from our theoretical formalism. For high-dimensional problems, we parameterize the policy using a neural network and optimize it based on an ergodic approximation of the cost.
Title:Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
Abstract:We present a geometric framework for Reinforcement Learning (RL) that views policies as maps into the Wasserstein space of action probabilities. First, we define a Riemannian structure induced by stationary distributions, proving its existence in a general context. We then define the tangent space of policies and characterize the geodesics, specifically addressing the measurability of vector fields mapped from the state space to the tangent space of probability measures over the action space. Next, we formulate a general RL optimization problem and construct a gradient flow using Otto's calculus. We compute the gradient and the Hessian of the energy, providing a formal second-order analysis. Finally, we illustrate the method with numerical examples for low-dimensional problems, computing the gradient directly from our theoretical formalism. For high-dimensional problems, we parameterize the policy using a neural network and optimize it based on an ergodic approximation of the cost.
Machine Learning (cs.LG); Optimization and Control (math.OC); Probability (math.PR)
arXiv:2604.14765 [cs.LG]
(or arXiv:2604.14765v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.14765 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 255 words
Related coverage
Non-Prehensile Throwing: A Reinforcement Learning Perspective
arXiv · Sep 2, 2026
Contextual Distributionally Robust Optimization with Causal and Continuous Structure
arXiv · Sep 30, 2026
Active Inference as a Convex Markov Decision Process
arXiv · Jul 23, 2026
Reinforcement Learning for Flow-Matching Policies with Density Transport
arXiv · Jun 9, 2026