aiqbench analysis
SimToolReal advances robot dexterity by using procedural simulation to train a single reinforcement learning policy capable of zero-shot manipulation across diverse tools and tasks. This approach eliminates the need for task-specific engineering or reward tuning, significantly improving generalization in sim-to-real transfer for complex grasping and forceful interactions.
SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation
arXiv:2602.16863v3 Announce Type: replace-cross Abstract: The ability to manipulate tools significantly expands the set of tasks a robot can perform. Yet, tool manipulation represents a challenging class of dexterity, requiring grasping thin objects, in-hand object rotations, and forceful interactions. Since collecting teleoperation data for these behaviors is challenging, sim-to-real reinforcement learning (RL) is a promising alternative. However, prior approaches typically require substantial engineering effort to model objects and tune reward functions for each task. In this work, we propose SimToolReal, taking a step towards generalizing sim-to-real RL policies for tool manipulation. Instead of focusing on a single object and task, we procedurally generate a large variety of tool-like object primitives in simulation and train a single RL policy with the universal goal of manipulating each object to random goal poses. This approach enables SimToolReal to perform general dexterous tool manipulation at test-time without any object or task-specific training. We demonstrate that SimToolReal outperforms prior retargeting and fixed-grasp methods by 37% while matching the performance of specialist RL policies trained on specific target objects and tasks. Finally, we show that SimToolReal generalizes across a diverse set of everyday tools, achieving strong zero-shot performance over 120 real-world rollouts spanning 24 tasks, 12 object instances, and 6 tool categories.
Abstract:The ability to manipulate tools significantly expands the set of tasks a robot can perform. Yet, tool manipulation represents a challenging class of dexterity, requiring grasping thin objects, in-hand object rotations, and forceful interactions. Since collecting teleoperation data for these behaviors is challenging, sim-to-real reinforcement learning (RL) is a promising alternative. However, prior approaches typically require substantial engineering effort to model objects and tune reward functions for each task. In this work, we propose SimToolReal, taking a step towards generalizing sim-to-real RL policies for tool manipulation. Instead of focusing on a single object and task, we procedurally generate a large variety of tool-like object primitives in simulation and train a single RL policy with the universal goal of manipulating each object to random goal poses. This approach enables SimToolReal to perform general dexterous tool manipulation at test-time without any object or task-specific training. We demonstrate that SimToolReal outperforms prior retargeting and fixed-grasp methods by 37% while matching the performance of specialist RL policies trained on specific target objects and tasks. Finally, we show that SimToolReal generalizes across a diverse set of everyday tools, achieving strong zero-shot performance over 120 real-world rollouts spanning 24 tasks, 12 object instances, and 6 tool categories.
23 pages, 16 figures, 3 tables. Project page: this https URL
Robotics (cs.RO); Artificial Intelligence (cs.AI)
arXiv:2602.16863 [cs.RO]
(or arXiv:2602.16863v3 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2602.16863 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 306 words
Related coverage
Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation
arXiv · Oct 14, 2026
Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction
arXiv · Sep 9, 2026
TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation
arXiv · Jun 16, 2026
DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real
arXiv · Sep 21, 2026