aiqbench analysis
Canary presents a significant advancement in risk-averse reinforcement learning by utilizing Cantelli's inequality to create stable, tractable bounds for Value-at-Risk constraints. This approach extends the Constrained Policy Optimization framework with worst-case guarantees that ensure reliable constraint satisfaction even in dense cost regimes where other methods fail. The research offers a robust solution for deploying RL agents in safety-critical applications where managing tail risk is paramount.
Cantelli Constrained Policy Optimization
arXiv:2601.22993v5 Announce Type: replace Abstract: We introduce Canary, a risk-averse method designed to optimize Value-at-Risk (VaR) constrained reinforcement learning (RL) problems. We employ Cantelli's inequality to obtain a tractable, conservative and smooth bound on the VaR constraint based on the first two moments of the cost return. This yields a constraint estimator that remains stable with tight violation thresholds in dense cost regimes. Extending the trust-region framework of the Constrained Policy Optimization (CPO) method, we further provide worst-case bounds for both policy improvement and constraint violation during the training process. Empirically during training, Canary is the only method that reliably satisfies the VaR constraint in every environment tested.
Abstract:We introduce Canary, a risk-averse method designed to optimize Value-at-Risk (VaR) constrained reinforcement learning (RL) problems. We employ Cantelli's inequality to obtain a tractable, conservative and smooth bound on the VaR constraint based on the first two moments of the cost return. This yields a constraint estimator that remains stable with tight violation thresholds in dense cost regimes. Extending the trust-region framework of the Constrained Policy Optimization (CPO) method, we further provide worst-case bounds for both policy improvement and constraint violation during the training process. Empirically during training, Canary is the only method that reliably satisfies the VaR constraint in every environment tested.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2601.22993 [cs.LG]
(or arXiv:2601.22993v5 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2601.22993 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 198 words
Related coverage
Optimal Constrained sc-LTL Planning in MDPs via Switching Policies
arXiv · Aug 6, 2026
Constrained Group Relative Policy Optimization
arXiv · Sep 2, 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Near-Optimal Reinforcement Learning for Constrained Recurrence Objectives
arXiv · Aug 5, 2026