Improving Policy Learning from Delayed Interventions

arXiv:2610.17298v1 Announce Type: new Abstract: In this work, we explore the idea of expert intervention delays in human-supervised robotic systems for imitation learning system. Previous works in imitation learning have primarily examined the impact of suboptimal experts, where the expert policy makes mistakes, but assumed expert interventions occur immediately after errors are made. Although delayed expert intervention can be viewed as a form of noise that might improve policy robustness, our results show that learning from delayed expert interventions is overall less effective than learning from non-delayed expert interventions; delayed expert interventions are less effective at demonstrating error avoidance behaviors. To address this, we propose Learning from Delayed Intervention (LfDI), which learns a cost function based on where and when the expert intervenes, combined with limited information about the extent of intervention delays provided by the expert. This cost function is then used to infer additional supervision for states without expert data, guiding the policy to avoid states that would likely trigger interventions. We evaluate our method, Learning from Delayed Intervention (LfDI), on simulated environments and show that it outperforms other methods that do not account for intervention delays.

Abstract:In this work, we explore the idea of expert intervention delays in human-supervised robotic systems for imitation learning system. Previous works in imitation learning have primarily examined the impact of suboptimal experts, where the expert policy makes mistakes, but assumed expert interventions occur immediately after errors are made. Although delayed expert intervention can be viewed as a form of noise that might improve policy robustness, our results show that learning from delayed expert interventions is overall less effective than learning from non-delayed expert interventions; delayed expert interventions are less effective at demonstrating error avoidance behaviors. To address this, we propose Learning from Delayed Intervention (LfDI), which learns a cost function based on where and when the expert intervenes, combined with limited information about the extent of intervention delays provided by the expert. This cost function is then used to infer additional supervision for states without expert data, guiding the policy to avoid states that would likely trigger interventions. We evaluate our method, Learning from Delayed Intervention (LfDI), on simulated environments and show that it outperforms other methods that do not account for intervention delays.

Robotics (cs.RO)

arXiv:2610.17298 [cs.RO]

(or arXiv:2610.17298v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2610.17298 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Improving Policy Learning from Delayed Interventions

Subiecte conexe