ResCue: Residual Spatial Cueing for Language-Conditioned Imitation Learning

Abstract:Visual imitation policies must learn which image regions matter for action, whether applying the same operation to different objects or selecting a language-specified target. We introduce ResCue (Residual Spatial Cueing), which converts language-derived target information into a spatial condition for visual policy learning. Pretrained grounding produces a target mask; an independent cue encoder adds its features to spatial RGB features before further visual processing and pooling. This retains scene context while making the target explicit. With Diffusion Policy (DP), ResCue achieves 56.5% average success across 15 simulated tasks, versus 33.1% for DP and 40.% for the strongest baseline, $\pi_0$, with gains in both ordinary manipulation and instruction-dependent object selection. On three real-world bimanual tasks, it reaches 60.0%, compared with 31.7% for $\pi_0$. Matched-mask ablations favor residual fusion over input-level and pooled alternatives, while test-time interventions show that target-correct guidance matters for execution.

Robotics (cs.RO)

arXiv:2606.25360 [cs.RO]

(or arXiv:2606.25360v2 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2606.25360 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.