The Geometry of Learning to Avoid Interventions

arXiv:2602.03825v2 Announce Type: replace Abstract: Human interventions are a common source of supervision in autonomous systems during deployment. Many existing approaches are based on avoiding interventions, yet the consequences of this objective are not well understood. We develop a geometric perspective on intervention learning that characterizes intervention avoidance as constraining policies to a face of the occupancy measure polytope. This view reveals that the effectiveness of intervention learning depends on the informativeness of the intervention strategy: highly informative interventions uniquely determine the solution, while weak interventions leave a large set of feasible policies, many of which are suboptimal. Motivated by this under-specification, we define Robust Intervention Learning (RIL) as the problem of learning policies that perform well under varying levels of intervention informativeness. From the geometric formulation, we derive Residual Intervention Fine-Tuning (RIFT), which combines interventions with a prior policy to select among feasible solutions. We show that RIFT provides provable improvement over the prior and corresponds to solving a constrained reinforcement learning problem with an induced reward. Empirically, RIFT yields consistent policy improvement across a range of intervention settings, particularly when interventions are sparse or weakly informative. These results highlight the importance of accounting for intervention informativeness and suggest a principled path toward robust learning from human feedback.

Abstract:Human interventions are a common source of supervision in autonomous systems during deployment. Many existing approaches are based on avoiding interventions, yet the consequences of this objective are not well understood. We develop a geometric perspective on intervention learning that characterizes intervention avoidance as constraining policies to a face of the occupancy measure polytope. This view reveals that the effectiveness of intervention learning depends on the informativeness of the intervention strategy: highly informative interventions uniquely determine the solution, while weak interventions leave a large set of feasible policies, many of which are suboptimal. Motivated by this under-specification, we define Robust Intervention Learning (RIL) as the problem of learning policies that perform well under varying levels of intervention informativeness. From the geometric formulation, we derive Residual Intervention Fine-Tuning (RIFT), which combines interventions with a prior policy to select among feasible solutions. We show that RIFT provides provable improvement over the prior and corresponds to solving a constrained reinforcement learning problem with an induced reward. Empirically, RIFT yields consistent policy improvement across a range of intervention settings, particularly when interventions are sparse or weakly informative. These results highlight the importance of accounting for intervention informativeness and suggest a principled path toward robust learning from human feedback.

Machine Learning (cs.LG)

arXiv:2602.03825 [cs.LG]

(or arXiv:2602.03825v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2602.03825 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

The Geometry of Learning to Avoid Interventions

Related coverage