aiqbench analysis
This research advances safety-critical reinforcement learning by addressing peak-cost constraints where single catastrophic failures must be avoided, distinguishing it from standard cumulative cost models. The authors introduce a robust surrogate optimization framework that mitigates simulator-to-real-world gaps while proving theoretical bounds on constraint violations under dynamics perturbations. These findings are significant for deploying RL agents in high-stakes environments like autonomous systems or robotics where strict safety limits cannot be breached even momentarily.
Robust Peak-cost Constrained Reinforcement Learning
arXiv:2607.15457v1 Announce Type: new Abstract: We study robust peak-cost constrained reinforcement learning (RP-CRL), where the objective is to maximize expected reward while controlling the maximum cost encountered along a trajectory. This setting is motivated by safety-critical applications in which a single large violation can be catastrophic and therefore cannot be adequately captured by the standard CMDP framework based on expected cumulative cost. Existing reachability-constrained RL methods adopt Lagrangian-based approaches, yet the underlying duality properties of peak-cost constrained MDPs remain unclear. We show that, unlike standard CMDPs, peak-cost constrained MDPs may not admit zero duality gap. We further consider a robust formulation to address simulator-to-real-world mismatch in the transition dynamics. To solve this problem, we develop a surrogate optimization framework and a robust value estimation method based on integral probability metrics. We prove that, with appropriate hyperparameter choices, the surrogate solution attains the same robust reward value as the original problem while violating the constraint by at most epsilon. Experiments show that the proposed method effectively enforces safety under dynamics perturbations while retaining strong reward performance.
Abstract:We study robust peak-cost constrained reinforcement learning (RP-CRL), where the objective is to maximize expected reward while controlling the maximum cost encountered along a trajectory. This setting is motivated by safety-critical applications in which a single large violation can be catastrophic and therefore cannot be adequately captured by the standard CMDP framework based on expected cumulative cost. Existing reachability-constrained RL methods adopt Lagrangian-based approaches, yet the underlying duality properties of peak-cost constrained MDPs remain unclear. We show that, unlike standard CMDPs, peak-cost constrained MDPs may not admit zero duality gap. We further consider a robust formulation to address simulator-to-real-world mismatch in the transition dynamics. To solve this problem, we develop a surrogate optimization framework and a robust value estimation method based on integral probability metrics. We prove that, with appropriate hyperparameter choices, the surrogate solution attains the same robust reward value as the original problem while violating the constraint by at most epsilon. Experiments show that the proposed method effectively enforces safety under dynamics perturbations while retaining strong reward performance.
Machine Learning (cs.LG)
arXiv:2607.15457 [cs.LG]
(or arXiv:2607.15457v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.15457 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 264 words
Related coverage
Robust Peak-cost Constrained Reinforcement Learning
arXiv · Sep 30, 2026
Reinforcement Learning on Cost-Constrained Quadrupedal Hardware
arXiv · Jul 30, 2026
RAVEN: Reinforcement-Adaptive Visibility-Graph Planning for Robust Humanoid Navigation with Collision-Free MPC
arXiv · Jul 20, 2026
SafeDiSTL: Decomposing Joint Constraints for Safe Multi-Agent Reinforcement Learning
arXiv · Oct 12, 2026