aiqbench analysis
This paper introduces Structural Dual Adaptation (SDLA), a novel method for safe reinforcement learning that stabilizes primal-dual optimization by replacing oscillatory bang-bang responses with continuous adjustments. Experimental results on OmniSafe and HASARD benchmarks demonstrate that SDLA achieves higher mean rewards than human baselines while strictly adhering to safety cost constraints, marking a significant advancement in constrained policy optimization.
Structural Dual Adaptation for Safe Reinforcement Learning
From arXiv
Full textarXiv:2611.00636v1 Announce Type: new Abstract: Primal-dual methods enforce safety constraints in reinforcement learning through Lagrangian relaxation. For a fixed policy, the exact bounded multiplier response exhibits bang-bang switching at the constraint boundary. Practical gradient updates can oscillate through coupling with policy optimization. We propose Structural DuaL Adaptation (SDLA), which replaces this bang-bang response with a unique continuous response through current-cost centering, trajectory weighting, and quadratic regularization. We characterize activation thresholds and response sensitivity.
Ternary search computes multipliers without a dual gradient stepsize. For entropy-regularized tabular policies, we bound equilibrium constraint violations alongside reward gaps. We establish convergence of delayed updates under mild assumptions. Neural PPO/SAC experiments on OmniSafe and HASARD demonstrate competitive rewards with improved constraint tracking. On HASARD, PPO-SDLA achieves higher reported mean reward than the human baseline in six of eighteen settings while meeting the respective mean-cost budgets, including four with lower mean cost. To the best of our knowledge, this is the first report of higher mean reward at lower mean cost than the human baseline across multiple HASARD settings.
Abstract:Primal-dual methods enforce safety constraints in reinforcement learning through Lagrangian relaxation. For a fixed policy, the exact bounded multiplier response exhibits bang-bang switching at the constraint boundary. Practical gradient updates can oscillate through coupling with policy optimization. We propose Structural DuaL Adaptation (SDLA), which replaces this bang-bang response with a unique continuous response through current-cost centering, trajectory weighting, and quadratic regularization. We characterize activation thresholds and response sensitivity. Ternary search computes multipliers without a dual gradient stepsize.
For entropy-regularized tabular policies, we bound equilibrium constraint violations alongside reward gaps. We establish convergence of delayed updates under mild assumptions. Neural PPO/SAC experiments on OmniSafe and HASARD demonstrate competitive rewards with improved constraint tracking. On HASARD, PPO-SDLA achieves higher reported mean reward than the human baseline in six of eighteen settings while meeting the respective mean-cost budgets, including four with lower mean cost. To the best of our knowledge, this is the first report of higher mean reward at lower mean cost than the human baseline across multiple HASARD settings.
Machine Learning (cs.LG)
arXiv:2611.00636 [cs.LG]
(or arXiv:2611.00636v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.00636 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 262 words
Related coverage
Rubrics Beyond Evaluation: Exploiting Rubric Structure in Reinforcement Learning
arXiv · Oct 21, 2026
Single or Multiple Policies for Phase-Structured Reinforcement Learning?
arXiv · Oct 6, 2026
Markovian Nonconvex ADMM for Reinforcement Learning: Bellman-Resolvent Stability Beyond Smooth Blocks
arXiv · Sep 30, 2026
Structure-aware Reinforcement Learning for Protein Directed Evolution
arXiv · Oct 1, 2026