aiqbench analysis
VibeAct advances robotic dexterity by using piezoelectric microphones to capture vibro-acoustic signals, enabling a shared tactile representation that bridges the gap between physical reality and simulation. This approach allows reinforcement learning policies to leverage rapid contact feedback without simulating raw audio, significantly improving success rates on complex manipulation tasks. The framework demonstrates a practical solution for handling fast, local interactions that are often invisible to standard visual sensors.
VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity
arXiv:2606.27344v2 Announce Type: replace Abstract: Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones provide compact, high-bandwidth sensing of these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared compact tactile representation. Guided by the microphones' sensitivity to contact transients and sliding-induced vibrations, we define this representation as binary contact onset, binary slip presence, and scalar slip magnitude for each fingertip. In the real world, we embed piezoelectric microphones in a dexterous robot hand and collect vibro-acoustic signals synchronized with robot and object states during teleoperation. We replay the recorded states in a calibrated digital clone to generate per-finger contact and slip labels for training a tactile estimator to infer the tactile representation from real microphone signals. In simulation, manipulation policies are trained using the same representation computed directly from simulated contact dynamics. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across eight contact-rich manipulation tasks, VibeAct improves average success from 21.6% to 50.5% over a baseline using proprioception and point clouds in simulation. The transferred policies also outperform the baseline on all three tasks evaluated on a physical dexterous hand-arm platform. Project videos and additional details are at https://vibeact.github.io/.
Abstract:Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones provide compact, high-bandwidth sensing of these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared compact tactile representation. Guided by the microphones' sensitivity to contact transients and sliding-induced vibrations, we define this representation as binary contact onset, binary slip presence, and scalar slip magnitude for each fingertip. In the real world, we embed piezoelectric microphones in a dexterous robot hand and collect vibro-acoustic signals synchronized with robot and object states during teleoperation. We replay the recorded states in a calibrated digital clone to generate per-finger contact and slip labels for training a tactile estimator to infer the tactile representation from real microphone signals. In simulation, manipulation policies are trained using the same representation computed directly from simulated contact dynamics. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across eight contact-rich manipulation tasks, VibeAct improves average success from 21.6% to 50.5% over a baseline using proprioception and point clouds in simulation. The transferred policies also outperform the baseline on all three tasks evaluated on a physical dexterous hand-arm platform. Project videos and additional details are at this https URL.
Robotics (cs.RO)
arXiv:2606.27344 [cs.RO]
(or arXiv:2606.27344v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2606.27344 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 317 words
Related coverage
Action Chunking Proximal Policy Optimization with Feedback Correction
arXiv · Sep 30, 2026
Beyond Dexterity: Why Contact May Define the Next Era of Robotics
IEEE Spectrum · Jun 9, 2026
TacOT: Learning Contact-Rich Dexterous Manipulation from Human Demonstrations via Tactile-Guided Optimal Transport
arXiv · Oct 6, 2026
ReTouch: Empowering Contact-Rich Dexterous Manipulation with Online-Refined Tactile Prediction
arXiv · Aug 4, 2026