Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

arXiv:2610.32125v1 Announce Type: cross Abstract: Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function $Q^*$ from a fixed dataset without further interaction with the environment. Although $Q^*$ is characterized by the soft optimal Bellman equation, the corresponding Bellman operator cannot be evaluated directly because both the reward distribution and the transition kernel are unknown. Consequently, estimating $Q^*$ requires the joint treatment of environment modeling and value function learning. We propose a framework that addresses these tasks separately. Conditional diffusion models based on probability flow ODEs estimate the conditional reward distribution and transition kernel. The resulting distributional estimators define a data driven approximation of the soft optimal Bellman operator, whose empirical Bellman residual is minimized over a class of deep neural networks to estimate $Q^*$. We establish nonasymptotic guarantees for the proposed method. For operator learning, we derive total variation error bounds for the conditional flow estimators that account explicitly for score estimation, numerical discretization, and terminal smoothing. For soft value learning, we first use the soft Bellman equation to show that $Q^*$ inherits H\"older regularity from the conditional reward and transition densities. For any H\"older index $\beta$, the value learning stage achieves an excess Bellman residual risk bound of order $\widetilde{\mathcal O}\big(n^{-\frac{2\beta}{d+2\beta}}\big),$ where $d$ is the total dimension of the state and action spaces. In contrast to many existing theoretical analyses of deep reinforcement learning, our results do not require Bellman completeness. Numerical experiments support the theoretical findings and demonstrate the effectiveness of the proposed method.

Abstract:Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function $Q^*$ from a fixed dataset without further interaction with the environment. Although $Q^*$ is characterized by the soft optimal Bellman equation, the corresponding Bellman operator cannot be evaluated directly because both the reward distribution and the transition kernel are unknown. Consequently, estimating $Q^*$ requires the joint treatment of environment modeling and value function learning. We propose a framework that addresses these tasks separately. Conditional diffusion models based on probability flow ODEs estimate the conditional reward distribution and transition kernel. The resulting distributional estimators define a data driven approximation of the soft optimal Bellman operator, whose empirical Bellman residual is minimized over a class of deep neural networks to estimate $Q^*$. We establish nonasymptotic guarantees for the proposed method. For operator learning, we derive total variation error bounds for the conditional flow estimators that account explicitly for score estimation, numerical discretization, and terminal smoothing. For soft value learning, we first use the soft Bellman equation to show that $Q^*$ inherits Hölder regularity from the conditional reward and transition densities. For any Hölder index $\beta$, the value learning stage achieves an excess Bellman residual risk bound of order $\widetilde{\mathcal O}\big(n^{-\frac{2\beta}{d+2\beta}}\big),$ where $d$ is the total dimension of the state and action spaces. In contrast to many existing theoretical analyses of deep reinforcement learning, our results do not require Bellman completeness. Numerical experiments support the theoretical findings and demonstrate the effectiveness of the proposed method.

Machine Learning (stat.ML); Machine Learning (cs.LG)

arXiv:2610.32125 [stat.ML]

(or arXiv:2610.32125v1 [stat.ML] for this version)

https://doi.org/10.48550/arXiv.2610.32125 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

Related coverage