aiqbench analysis
M3imic addresses a critical gap in humanoid robotics by unifying heterogeneous motion reference modalities—such as joint angles, human poses, and end-effector trajectories—into a single shared latent space. By leveraging large-scale reinforcement learning, the framework enables a single policy to achieve sim-to-real transfer across these diverse inputs without requiring modality-specific retraining. This approach significantly enhances generalization capabilities for whole-body control tasks, as demonstrated by a 98.42% success rate on unseen datasets in simulation.
M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking
arXiv:2606.04829v1 Announce Type: new Abstract: Building a general-purpose whole-body controller is essential for enabling diverse motion capabilities in humanoid robots across a wide range of downstream tasks, including locomotion and loco-manipulation. Different tasks rely on distinct motion reference modalities: locomotion primarily depends on coordinated robot joint trajectories, whereas manipulation requires precise end-effector trajectory tracking. Existing methods often overlook the representational mismatch between dense robot joint angles and sparse end-effector poses. To address this, we propose Multi-Modal Mimic (M3imic), a versatile multi-modal whole-body control framework that unifies heterogeneous motion reference modalities, including robot joint angles, human pose trajectories, and end-effector poses, using modality-specific encoders to map them into a shared latent space. Leveraging large-scale reinforcement learning in the simulator, we train a single policy that achieves sim-to-real transfer across multiple motion reference modalities without modality-specific retraining. Extensive simulation and real-world experiments on the Unitree G1 robot are conducted to evaluate the proposed framework. In simulation, the policy achieves a peak success rate of 98.42\% on an unseen test dataset, demonstrating its exceptional generalization capability. The code is available at https://github.com/Renforce-Dynamics/MultiModalWBC
Abstract:Building a general-purpose whole-body controller is essential for enabling diverse motion capabilities in humanoid robots across a wide range of downstream tasks, including locomotion and loco-manipulation. Different tasks rely on distinct motion reference modalities: locomotion primarily depends on coordinated robot joint trajectories, whereas manipulation requires precise end-effector trajectory tracking. Existing methods often overlook the representational mismatch between dense robot joint angles and sparse end-effector poses. To address this, we propose Multi-Modal Mimic (M3imic), a versatile multi-modal whole-body control framework that unifies heterogeneous motion reference modalities, including robot joint angles, human pose trajectories, and end-effector poses, using modality-specific encoders to map them into a shared latent space. Leveraging large-scale reinforcement learning in the simulator, we train a single policy that achieves sim-to-real transfer across multiple motion reference modalities without modality-specific retraining. Extensive simulation and real-world experiments on the Unitree G1 robot are conducted to evaluate the proposed framework. In simulation, the policy achieves a peak success rate of 98.42\% on an unseen test dataset, demonstrating its exceptional generalization capability. The code is available at this https URL
Robotics (cs.RO)
arXiv:2606.04829 [cs.RO]
(or arXiv:2606.04829v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2606.04829 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 269 words
Related coverage
ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation
arXiv · Sep 21, 2026
ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Data
arXiv · Jun 4, 2026
iGPC: Generative Motion Priors for Object-Aware Humanoid Interaction
arXiv · Oct 7, 2026
Light-Loco-Parkour: Versatile Perceptive Whole-Body Locomotion via Multi-Skill Distillation
arXiv · Aug 5, 2026