analiza aiqbench
This research advances robotic manipulation by aligning visual and tactile data through contrastive learning, significantly boosting task success rates in contact-rich environments. By integrating these multi-modal representations into an Action Chunking Transformer policy, the study demonstrates a practical pathway for robots to better interpret physical feedback alongside standard vision inputs. The development of a low-cost sensing platform further supports reproducible data collection for training robust manipulation algorithms.
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
arXiv:2602.00514v3 Announce Type: replace Abstract: Tactile feedback is important for contact-rich robotic manipulation, yet effective use of tactile observations remains challenging when tactile signals are image-like, hardware-dependent, and only weakly aligned with external visual observations. This study addresses this representation-learning problem by proposing a visuo-tactile contrastive learning framework for imitation-based manipulation. The method aligns external RGB observations and calibrated tactile images in a shared embedding space using a CLIP-style objective, and integrates the resulting representation into an Action Chunking Transformer (ACT) policy. A low-cost visuo-tactile gripper (LVTG) is proposed to provide a modular and durable sensing platform for reproducible data collection, supplying tactile observations that can be used by downstream manipulation algorithms. Experiments on contact-rich manipulation tasks show that tactile feedback improves the average task completion rate from 30% for a vision-only ACT baseline to 42%, and that the proposed contrastive pretraining further increases the completion rate to 54%. These results indicate that explicitly aligning visual and tactile observations provides more useful contact-aware features for downstream policy learning than directly adding tactile images without pretraining.
Abstract:Tactile feedback is important for contact-rich robotic manipulation, yet effective use of tactile observations remains challenging when tactile signals are image-like, hardware-dependent, and only weakly aligned with external visual observations. This study addresses this representation-learning problem by proposing a visuo-tactile contrastive learning framework for imitation-based manipulation. The method aligns external RGB observations and calibrated tactile images in a shared embedding space using a CLIP-style objective, and integrates the resulting representation into an Action Chunking Transformer (ACT) policy. A low-cost visuo-tactile gripper (LVTG) is proposed to provide a modular and durable sensing platform for reproducible data collection, supplying tactile observations that can be used by downstream manipulation algorithms. Experiments on contact-rich manipulation tasks show that tactile feedback improves the average task completion rate from 30% for a vision-only ACT baseline to 42%, and that the proposed contrastive pretraining further increases the completion rate to 54%. These results indicate that explicitly aligning visual and tactile observations provides more useful contact-aware features for downstream policy learning than directly adding tactile images without pretraining.
Robotics (cs.RO)
arXiv:2602.00514 [cs.RO]
(or arXiv:2602.00514v3 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2602.00514 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 261 cuvinte
Subiecte conexe
Disentangling Visuo-Tactile Foresight: Oracle-Guided Interface Discovery for World Action Models
arXiv · 4 aug. 2026
CAAT: Contact-Aware Attention Scaling and Tactile Masking for Data-Efficient Contact-Rich Manipulation
arXiv · 4 aug. 2026
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
arXiv · 8 oct. 2026
RGB-S: Image-Aligned Tactile Saliency for Robust Dexterous Manipulation
arXiv · 12 iun. 2026