Inside NVIDIA’s IsaacTeleop: From Hand and Controller Tracking to Robot Actions with the Graph-Based Retargeting Engine

In this tutorial, we work through the retargeting engine at the core of NVIDIA IsaacTeleop, the framework that turns XR hand tracking and motion-controller input into commands for simulated and real robots. Rather than plugging in a headset, we build every input ourselves in NumPy, so each step runs on a plain Colab CPU and prints what it computes. We start with the type system every node speaks, generate synthetic hand and controller data, write our own retargeter with live-tunable parameters, and then drive the built-in gripper and SE(3) retargeters with it. From there, we compose a full graph that emits one action vector per step, apply a world-frame transform, step through the run, pause and kill the state machine, and finish with a controller-to-dexterous-hand mapping and parameter tuning that persists across restarts. We install the…