analiza aiqbench
This paper introduces MultiSensory Dynamic Pretraining (MSDP), a framework that uses masked autoencoding to fuse vision, force, and proprioception data for robust robot manipulation. By employing an asymmetric architecture where the critic accesses dynamic features while the actor relies on stable embeddings, the method achieves high success rates with only 6,000 online interactions despite sensor noise. The approach offers a practical solution for scaling reinforcement learning in complex, contact-rich environments through efficient cross-modal representation learning.
Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
arXiv:2511.14427v4 Announce Type: replace-cross Abstract: Effective contact-rich manipulation requires robots to synergistically leverage vision, force, and proprioception. However, Reinforcement Learning agents struggle to learn in such multisensory settings, especially amidst sensory noise and dynamic changes. We propose MultiSensory Dynamic Pretraining (MSDP), a novel framework for learning expressive multisensory representations tailored for task-oriented policy learning. MSDP is based on masked autoencoding and trains a transformer-based encoder by reconstructing multisensory observations from only a subset of sensor embeddings, leading to cross-modal prediction and sensor fusion. For downstream policy learning, we introduce a novel asymmetric architecture, where a cross-attention mechanism allows the critic to extract dynamic, task-specific features from the frozen embeddings, while the actor receives a stable pooled representation to guide its actions. Our method demonstrates accelerated learning and robust performance under diverse perturbations, including sensor noise, and changes in object dynamics. Evaluations in multiple challenging, contact-rich robot manipulation tasks in simulation and the real world showcase the effectiveness of MSDP. Our approach exhibits strong robustness to perturbations and achieves high success rates on the real robot with as few as 6,000 online interactions, offering a simple yet powerful solution for complex multisensory robotic control. Website: https://msdp-pearl.github.io/
Abstract:Effective contact-rich manipulation requires robots to synergistically leverage vision, force, and proprioception. However, Reinforcement Learning agents struggle to learn in such multisensory settings, especially amidst sensory noise and dynamic changes. We propose MultiSensory Dynamic Pretraining (MSDP), a novel framework for learning expressive multisensory representations tailored for task-oriented policy learning. MSDP is based on masked autoencoding and trains a transformer-based encoder by reconstructing multisensory observations from only a subset of sensor embeddings, leading to cross-modal prediction and sensor fusion. For downstream policy learning, we introduce a novel asymmetric architecture, where a cross-attention mechanism allows the critic to extract dynamic, task-specific features from the frozen embeddings, while the actor receives a stable pooled representation to guide its actions. Our method demonstrates accelerated learning and robust performance under diverse perturbations, including sensor noise, and changes in object dynamics. Evaluations in multiple challenging, contact-rich robot manipulation tasks in simulation and the real world showcase the effectiveness of MSDP. Our approach exhibits strong robustness to perturbations and achieves high success rates on the real robot with as few as 6,000 online interactions, offering a simple yet powerful solution for complex multisensory robotic control. Website: this https URL
Robotics (cs.RO); Machine Learning (cs.LG)
arXiv:2511.14427 [cs.RO]
(or arXiv:2511.14427v4 [cs.RO] for this version)
IEEE Robotics and Automation Letters, 2026, Vol. 11, No. 6, pp. 6799-6806
https://doi.org/10.1109/LRA.2026.3681156 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 223 cuvinte
Subiecte conexe
TacCoRL: Integrating Tactile Feedback into VLA via Simulation
arXiv · 12 iun. 2026
Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
arXiv · 25 sept. 2026
Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning
arXiv · 12 iun. 2026
Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning
arXiv · 11 iun. 2026