aiqbench analysis
ViBe introduces a parameter-efficient framework that adapts pre-trained visual encoders to provide exteroceptive feedback for humanoid robots, enabling zero-shot sim-to-real transfer across diverse physical tasks like parkour and manipulation. By grafting task-relevant perceptive feedback onto existing motion trackers via low-rank adapters, this approach bridges the gap between scalable tracking methods and complex environmental interaction without requiring full retraining of geometry-only models. The method demonstrates robust performance in challenging conditions such as low light and outdoor settings, offering a practical pathway for deploying perceptive controllers on real-world hardware.
ViBe: Visual Behavior Adaptation for Perceptive Humanoid Whole-Body Control
arXiv:2609.09918v1 Announce Type: new Abstract: Motion tracking provides a scalable recipe for humanoid whole-body control. By design, the resulting trackers lack exteroceptive feedback hence reacting to the environment remains the responsibility of a higher-level planner. Existing perceptive controllers train geometry-only encoders from scratch, trading semantics for sim-to-real ease, and typically rely on teacher-student distillation for a task of interest. We present ViBe, a post-training framework for adapting motion trackers to perceptive control tasks. We leverage pre-trained visual encoders with a multi-query extractor module to learn task-relevant perceptive feedback. This feedback is grafted onto the tracker's input via low-rank adapters, enabling parameter-efficient fine-tuning. Given a task reward and a reference dataset, this modular controller can be adapted directly via policy optimization. Across four tasks, ViBe shows zero-shot sim-to-real transfer spanning perceptive walking on curbs and parkour, Repose Cube, omni-object loco-manipulation, and dodgeball, with visually robust performance across outdoor, low-light, and RGB distractor conditions. Finally, we solve a goal-oriented Repose Cube task with a deliberately simple planner, demonstrating the efficacy of perceptive controllers, adapted by our approach.
Abstract:Motion tracking provides a scalable recipe for humanoid whole-body control. By design, the resulting trackers lack exteroceptive feedback hence reacting to the environment remains the responsibility of a higher-level planner. Existing perceptive controllers train geometry-only encoders from scratch, trading semantics for sim-to-real ease, and typically rely on teacher-student distillation for a task of interest. We present ViBe, a post-training framework for adapting motion trackers to perceptive control tasks. We leverage pre-trained visual encoders with a multi-query extractor module to learn task-relevant perceptive feedback. This feedback is grafted onto the tracker's input via low-rank adapters, enabling parameter-efficient fine-tuning. Given a task reward and a reference dataset, this modular controller can be adapted directly via policy optimization. Across four tasks, ViBe shows zero-shot sim-to-real transfer spanning perceptive walking on curbs and parkour, Repose Cube, omni-object loco-manipulation, and dodgeball, with visually robust performance across outdoor, low-light, and RGB distractor conditions. Finally, we solve a goal-oriented Repose Cube task with a deliberately simple planner, demonstrating the efficacy of perceptive controllers, adapted by our approach.
Robotics (cs.RO)
arXiv:2609.09918 [cs.RO]
(or arXiv:2609.09918v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2609.09918 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics
Latent Space · Oct 13, 2026
Praxis: Distilling Physical Interaction Priors from Egocentric Videos for Generalizable Whole-Body Manipulation
arXiv · Sep 28, 2026
NEXUS: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation
arXiv · Oct 1, 2026
A System for Fast, Resilient, and Adaptable Loco-Manipulation Behaviors on Humanoid Robots
arXiv · Sep 2, 2026