Visibility-Aware Mobile Grasping in Dynamic Environments

Abstract:We consider mobile grasping in unknown and dynamic environments, where a robot must approach and grasp a target object using a camera with a limited field of view. This setting introduces two visibility constraints. Incomplete environmental observations can create collision risks along the whole-body trajectory, while occlusions can hinder object pose refinement during grasping. Previous systems often assume known or static environments and decouple perception from motion, leaving safety unresolved around unobserved obstacles. We present a unified system that enforces both visibility constraints at runtime through three core components. A velocity-aware gaze policy directs perception toward fast-moving regions of the planned swept volume, a hierarchical subgoal policy samples grasp and intermediate configurations that maintain target visibility, and a real-time whole-body planner replans over a receding horizon as the map updates. Across 400 randomized simulation scenarios and real-world trials on a Fetch mobile manipulator, our system achieves success rates of 70.4% in unknown static environments and 57.9% in dynamic environments, outperforming the decoupled navigation- and-manipulation strategy by 16.3 and 9.3 percentage points, respectively, while significantly reducing collisions.

Robotics (cs.RO)

arXiv:2605.02487 [cs.RO]

(or arXiv:2605.02487v4 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2605.02487 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.