Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation

Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso, Shuran Song

Seminal May 2025

Learned XHand policy on the long-horizon kitchen task, 2x speed.

TL;DR DexUMI records dexterous-hand demonstrations with an exoskeleton worn on the human hand that matches the robot hand's fingertip kinematics, then paints the robot hand over the human hand in the wrist-camera video. Policies trained only on this data average 86% success on four real tasks across two robot hands, and collection is 3.2x faster than teleoperation.

What they do

Teleoperating a dexterous hand gives no haptic feedback, and human hand video does not transfer directly to robot hands with different kinematics and appearance. In DexUMI a person does the task wearing an exoskeleton built for a specific robot hand, and the wrist-camera video is then edited so the robot hand appears in place of the human one. Data collection needs no robot arm.

How it works

  • Exoskeleton. Joint positions and link lengths are optimized so its fingertip workspace covers the robot’s and stays inside it, under bounds that keep it wearable.
  • Sensing. Encoders on every actuated joint, iPhone ARKit for wrist pose, the same under-wrist camera as the robot, and the same tactile sensors as the target hand.
  • Visual adaptation. SAM2 removes hand and exoskeleton, ProPainter fills the background, the actions are replayed on the robot hand alone, and only pixels inside both masks are pasted in, preserving occlusions.
  • Policy. A diffusion policy on DINOv2 features and tactile readings predicts relative wrist and finger actions.

Results

  • Full system, 20 trials per task: cube 1.00, egg carton 0.85 and tea 0.85 on the Inspire hand; tea 0.85 and the four-step kitchen task 0.75 on the XHand.
  • Absolute finger actions with tactile drop the cube to 0.10 and the kitchen task to 0.00.
  • Without inpainting, cube falls from 0.95 to 0.60 (green mask) or 0.20 (raw images), egg carton from 0.90 to 0.10 or 0.05.
  • Tactile lifts the XHand salt step from 0.15 to 0.75, but the noisier sensors added to the Inspire hand mostly hurt. DexUMI collects 3.2x more successful demos than teleoperation in 15-minute sessions.

Limitations

Each hand needs its own exoskeleton with hand-tuned wearability, and only fingertips are matched. Tactile sensors drift, and the 3D-printed links flex without the encoders noticing. Rendering still needs the real robot hand, the camera must be rigidly mounted, and inpainting misses lighting. Backlash in both hands limits the encoder-to-motor mapping.

Videos

Autonomous rollout picking up tweezers and transferring tea leaves to a cup, 2x speed.
Inspire Hand policy picking up a small cube and placing it in a cup, 2x speed.

Figures

From Figure 1: Demonstrations with the DexUMI exoskeleton and the matching robot executions on the underactuated Insp...
From Figure 1: Demonstrations with the DexUMI exoskeleton and the matching robot executions on the underactuated Inspire hand and the fully actuated XHand, across long-horizon, contact-rich, multi-finger and precise tasks.
From Figure 2: The optimized exoskeleton shares the target robot hand's joint-to-fingertip mapping while staying wear...
From Figure 2: The optimized exoskeleton shares the target robot hand's joint-to-fingertip mapping while staying wearable. Joint encoders record finger actions, a 150° wide-angle camera records observations, and a rigidly mounted iPhone tracks the wrist pose with ARKit.
From Figure 4: The exoskeleton is segmented with SAM2 and the background inpainted. The recorded joint actions are re...
From Figure 4: The exoskeleton is segmented with SAM2 and the background inpainted. The recorded joint actions are replayed on the robot hand and its mask extracted; the intersection of the two masks gives the visible hand pixels, which are pasted onto the inpainted background.
From Figure 6: a) Relative finger actions give more precise actions and better multi-finger coordination than absolut...
From Figure 6: a) Relative finger actions give more precise actions and better multi-finger coordination than absolute ones. b) Even with noisy readings, tactile input fixes the visually hard salt-picking step.
Abstract

We present DexUMI - a data collection and policy learning framework that uses the human hand as the natural interface to transfer dexterous manipulation skills to various robot hands. DexUMI includes hardware and software adaptations to minimize the embodiment gap between the human hand and various robot hands. The hardware adaptation bridges the kinematics gap using a wearable hand exoskeleton. It allows direct haptic feedback in manipulation data collection and adapts human motion to feasible robot hand motion. The software adaptation bridges the visual gap by replacing the human hand in video data with high-fidelity robot hand inpainting. We demonstrate DexUMI's capabilities through comprehensive real-world experiments on two different dexterous robot hand hardware platforms, achieving an average task success rate of 86%.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.