aiqbench analysis
SB-TRPO addresses a critical bottleneck in safety-critical reinforcement learning by enforcing strict, zero-cost safety constraints without sacrificing task performance. The method achieves this through a principled trust region update that balances maximal cost reduction with reward optimization, validated on the Safety Gymnasium benchmark. This research advances the field of safe RL by demonstrating convergence to optimal policies within feasible safety boundaries.
SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
arXiv:2512.23770v4 Announce Type: replace Abstract: In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce \emph{Safety-Biased Trust Region Policy Optimisation (SB-TRPO)}, a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on \emph{Safety Gymnasium} demonstrate high safety alongside strong task performance.
Abstract:In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce \emph{Safety-Biased Trust Region Policy Optimisation (SB-TRPO)}, a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on \emph{Safety Gymnasium} demonstrate high safety alongside strong task performance.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2512.23770 [cs.LG]
(or arXiv:2512.23770v4 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2512.23770 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 212 words
Related coverage
Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learning
arXiv · Sep 30, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
RL & search is a terrifying way to build AGI (an FAQ)
AI Alignment Forum · Jul 27, 2026
Constrained Goal-directed Planar Graph Generation with Grammar-based Reinforcement Learning
arXiv · Oct 6, 2026