analiza aiqbench
This paper addresses the non-uniqueness problem in classical inverse reinforcement learning by introducing a statistical framework that combines entropy regularization with least-squares reconstruction to derive a unique reward function. The authors establish non-asymptotic minimax optimal convergence rates, offering rigorous theoretical bounds on policy estimation error relative to sample size and model complexity. This work strengthens the mathematical foundations of IRL, bridging behavior cloning with modern statistical learning theory for more robust agent training.
Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
arXiv:2512.06956v2 Announce Type: replace-cross Abstract: Inverse reinforcement learning aims to infer the reward function that explains expert behavior observed through trajectories of state--action pairs. A long-standing difficulty in classical IRL is the non-uniqueness of the recovered reward: many reward functions can induce the same optimal policy, rendering the inverse problem ill-posed. In this paper, we develop a statistical framework for Inverse Entropy-regularized Reinforcement Learning that resolves this ambiguity by combining entropy regularization with a least-squares reconstruction of the reward from the soft Bellman residual. This combination yields a unique and well-defined so-called least-squares reward consistent with the expert policy. We model the expert demonstrations as a Markov chain with the invariant distribution defined by an unknown expert policy $\pi^\star$ and estimate the policy by a penalized maximum-likelihood procedure over a class of conditional distributions on the action space. We establish high-probability bounds for the excess Kullback--Leibler divergence between the estimated policy and the expert policy, accounting for statistical complexity through covering numbers of the policy class. These results lead to non-asymptotic minimax optimal convergence rates for the least-squares reward function, revealing the interplay between smoothing (entropy regularization), model complexity, and sample size. Our analysis bridges the gap between behavior cloning, inverse reinforcement learning, and modern statistical learning theory.
Abstract:Inverse reinforcement learning aims to infer the reward function that explains expert behavior observed through trajectories of state--action pairs. A long-standing difficulty in classical IRL is the non-uniqueness of the recovered reward: many reward functions can induce the same optimal policy, rendering the inverse problem ill-posed. In this paper, we develop a statistical framework for Inverse Entropy-regularized Reinforcement Learning that resolves this ambiguity by combining entropy regularization with a least-squares reconstruction of the reward from the soft Bellman residual. This combination yields a unique and well-defined so-called least-squares reward consistent with the expert policy. We model the expert demonstrations as a Markov chain with the invariant distribution defined by an unknown expert policy $\pi^\star$ and estimate the policy by a penalized maximum-likelihood procedure over a class of conditional distributions on the action space. We establish high-probability bounds for the excess Kullback--Leibler divergence between the estimated policy and the expert policy, accounting for statistical complexity through covering numbers of the policy class. These results lead to non-asymptotic minimax optimal convergence rates for the least-squares reward function, revealing the interplay between smoothing (entropy regularization), model complexity, and sample size. Our analysis bridges the gap between behavior cloning, inverse reinforcement learning, and modern statistical learning theory.
Machine Learning (stat.ML); Machine Learning (cs.LG); Statistics Theory (math.ST)
arXiv:2512.06956 [stat.ML]
(or arXiv:2512.06956v2 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2512.06956 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 301 cuvinte
Subiecte conexe
Exploding variance of means of exponentials: least-squares to the rescue – Machine Learning Research Blog
francisbach.com · 27 sept. 2026
On the Computational and Statistical Efficiency of the Empirical Maximum Entropy on the Mean Method
arXiv · 31 aug. 2026
Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models
arXiv · 28 iul. 2026
Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models
arXiv · 12 iun. 2026