Policy Gradients for Cumulative Prospect Theory in Reinforcement Learning

Abstract:We derive a policy gradient theorem for Cumulative Prospect Theory (CPT) objectives in finite-horizon Reinforcement Learning (RL), generalizing the standard policy gradient theorem and encompassing distortion-based risk objectives as special cases. Motivated by behavioral economics, CPT combines an asymmetric utility transformation around a reference point with probability distortion. Building on our theorem, we design a first-order policy gradient algorithm for CPT-RL using a Monte Carlo gradient estimator based on order statistics. We establish statistical guarantees for the estimator and prove asymptotic convergence of the resulting algorithm to first-order stationary points of the (generally nonconvex) CPT objective. We complement our asymptotic analysis with a non-asymptotic total sample complexity analysis to reach an approximate first-order stationary policy. Simulations illustrate qualitative behaviors induced by CPT and compare our first-order approach to existing zeroth-order methods.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2410.02605 [cs.LG]

(or arXiv:2410.02605v5 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2410.02605 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.