analiza aiqbench
This paper introduces Stochastic Decision Horizons (SDH), a novel framework for survival-based constrained reinforcement learning that dynamically adjusts decision horizons upon constraint violations to improve training efficiency. By deriving off-policy algorithms like VT-MPO and AS-SAC, the authors demonstrate significant reductions in environment steps required to achieve peak performance on complex humanoid locomotion tasks compared to existing methods. The work advances the theoretical understanding of geometric-horizon constraints and their relationship to cumulative cost functions within Constrained Markov Decision Processes (CMDPs).
Stochastic Decision Horizons for Survival-Based Constrained Reinforcement Learning
arXiv:2602.04599v3 Announce Type: replace Abstract: We propose stochastic decision horizons (SDH), a theoretically grounded framework for survival-based constrained reinforcement learning from per-step violation signals. Rather than treating violations as expenditure of a cumulative budget, SDH shortens the effective decision horizon when violations occur, reducing current reward credit and future bootstrapping. We show that this construction represents a geometric-horizon survival constraint and, for a fixed Lagrange multiplier, remains the same SDH problem up to a constant reward offset that acts as an implicit incentive for survival. Using control as inference, we derive two off-policy and replay-compatible, regularized algorithms with post-violation semantics. Absorbing-state ends the decision process, so only surviving decisions pay policy KL, yielding AS-SAC. Virtual termination stops reward credit while the process continues, retaining ordinary discounted KL and yielding VT-MPO. To connect SDH with cumulative-cost CMDPs, we introduce violation-depth profiles. Under exponential-tail single-scale profiles, the SDH statistic determines cumulative cost; departures from this form mark where the connection breaks. Experiments validate both the method and this predicted scope. On the 90-muscle H2190 humanoid in Hyfydy, VT-MPO matches the EWA reference in peak gait realism, with its highest observed gait-match checkpoint at 21M environment steps versus 90M for EWA and markedly more stable training. On Safety Gymnasium, violation-depth profiles predict the regimes in which SDH achieves strong reward-violation trade-offs.
Abstract:We propose stochastic decision horizons (SDH), a theoretically grounded framework for survival-based constrained reinforcement learning from per-step violation signals. Rather than treating violations as expenditure of a cumulative budget, SDH shortens the effective decision horizon when violations occur, reducing current reward credit and future bootstrapping. We show that this construction represents a geometric-horizon survival constraint and, for a fixed Lagrange multiplier, remains the same SDH problem up to a constant reward offset that acts as an implicit incentive for survival. Using control as inference, we derive two off-policy and replay-compatible, regularized algorithms with post-violation semantics. Absorbing-state ends the decision process, so only surviving decisions pay policy KL, yielding AS-SAC. Virtual termination stops reward credit while the process continues, retaining ordinary discounted KL and yielding VT-MPO. To connect SDH with cumulative-cost CMDPs, we introduce violation-depth profiles. Under exponential-tail single-scale profiles, the SDH statistic determines cumulative cost; departures from this form mark where the connection breaks. Experiments validate both the method and this predicted scope. On the 90-muscle H2190 humanoid in Hyfydy, VT-MPO matches the EWA reference in peak gait realism, with its highest observed gait-match checkpoint at 21M environment steps versus 90M for EWA and markedly more stable training. On Safety Gymnasium, violation-depth profiles predict the regimes in which SDH achieves strong reward-violation trade-offs.
Machine Learning (cs.LG)
arXiv:2602.04599 [cs.LG]
(or arXiv:2602.04599v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.04599 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 305 cuvinte
Subiecte conexe
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · 18 sept. 2026
Convex-Concave Reinforcement Learning
arXiv · 8 oct. 2026
Hard Constraints, Smooth Gradients: Learning Feasible Inventory Policies via Differentiable Projection
arXiv · 5 aug. 2026
The Alignment Problem: Computer Science Meets Analytic Philosophy
AI Alignment Forum · 18 oct. 2026