analiza aiqbench
This research introduces a distributionally robust Q-learning algorithm designed to handle model uncertainty in continuous state spaces by optimizing for worst-case transitions using Sinkhorn distance regularization. By establishing convergence guarantees and demonstrating effectiveness in portfolio optimization, the study advances the reliability of deep reinforcement learning under imperfect environmental models. The work provides critical theoretical foundations for deploying RL agents in high-stakes domains where distributional shifts could lead to significant failures.
Distributionally Robust Deep Q-Learning
arXiv:2505.19058v2 Announce Type: replace Abstract: We propose a novel distributionally robust $Q$-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterised with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S\&{P}~500 data. We also establish convergence guarantees for exact and approximate robust fitted $Q$-iteration, decompose the numerical RDQN error into interpretable components, and discuss extensions to compact continuous action sets.
Abstract:We propose a novel distributionally robust $Q$-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterised with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S\&{P}~500 data. We also establish convergence guarantees for exact and approximate robust fitted $Q$-iteration, decompose the numerical RDQN error into interpretable components, and discuss extensions to compact continuous action sets.
Machine Learning (cs.LG); Optimization and Control (math.OC); Portfolio Management (q-fin.PM); Machine Learning (stat.ML)
arXiv:2505.19058 [cs.LG]
(or arXiv:2505.19058v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2505.19058 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 256 cuvinte
