analiza aiqbench
This paper introduces CoDRA, a novel adversarial training framework for reinforcement learning that stabilizes controller performance against external disturbances without requiring auxiliary trade-off parameters. By optimizing the ratio of accumulated cost to disturbance norm, the method effectively moderates adversarial incentives while preserving nominal system performance in simulation-to-reality transfer scenarios. The approach demonstrates superior robustness across varying force and mass conditions in pendulum environments, addressing a critical gap in deploying RL policies on physical systems.
Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning
arXiv:2603.12110v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) policies trained in simulation often degrade once deployed on real systems, where the controller must reject external disturbances that were never encountered in simulation. Robust RL addresses this by exposing the controller to perturbations while it learns, through domain randomization, adversarial minimax formulations, or probabilistic mixtures of protagonist and adversarial behavior. However, an unregulated disturbance mechanism destabilizes training and often collapses nominal performance relative to standard, non-robust methods. We propose cost-to-disturbance ratio adversarial training (CoDRA), a framework that expresses the controller--adversary trade-off as a ratio of accumulated cost to accumulated squared disturbance norm, and optimizes it through a self-normalized actor--critic update. In this algorithm, each value term is scaled by a stop-gradient normalization constant computed from the current batch. This moderates the adversary's incentive without altering the controller's own update, and requires neither an explicit disturbance penalty nor an auxiliary trade-off parameter. We evaluate CoDRA on two MuJoCo pendulum environments under force and mass sweeps. On InvertedDoublePendulum, CoDRA attains the lowest cost at every force level, including a force outside the range seen during training, and in all but one cell of the mass grid, whereas its advantage is less pronounced on the milder InvertedPendulum.
Abstract:Reinforcement learning (RL) policies trained in simulation often degrade once deployed on real systems, where the controller must reject external disturbances that were never encountered in simulation. Robust RL addresses this by exposing the controller to perturbations while it learns, through domain randomization, adversarial minimax formulations, or probabilistic mixtures of protagonist and adversarial behavior. However, an unregulated disturbance mechanism destabilizes training and often collapses nominal performance relative to standard, non-robust methods. We propose cost-to-disturbance ratio adversarial training (CoDRA), a framework that expresses the controller--adversary trade-off as a ratio of accumulated cost to accumulated squared disturbance norm, and optimizes it through a self-normalized actor--critic update. In this algorithm, each value term is scaled by a stop-gradient normalization constant computed from the current batch. This moderates the adversary's incentive without altering the controller's own update, and requires neither an explicit disturbance penalty nor an auxiliary trade-off parameter. We evaluate CoDRA on two MuJoCo pendulum environments under force and mass sweeps. On InvertedDoublePendulum, CoDRA attains the lowest cost at every force level, including a force outside the range seen during training, and in all but one cell of the mass grid, whereas its advantage is less pronounced on the milder InvertedPendulum.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2603.12110 [cs.LG]
(or arXiv:2603.12110v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.12110 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 293 cuvinte
Subiecte conexe
Assessing Information Ecosystems : How Governments Can Get Ahead of Hybrid Threats
carnegieendowment.org · 5 aug. 2026
Reflection’s Beam model hits 501 billion parameters
Quantum Zeitgeist · 5 oct. 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · 13 oct. 2026
Fireside chat
ecb.europa.eu · 8 sept. 2026