DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection

Amber Xie, Jiawei Fu, Xiaofeng Guo, Carmelo Sferrazza, Guanya Shi, Haozhi Qi

October 2026

Screwing with in-hand tool rotation, autonomous.

TL;DR A hand-held rig that carries the robot's own 20-DoF hand and camera, so demonstrations need no retargeting. Simple joystick skills drive the fingers, and the commanded joint targets, not measured states, are logged as actions; that alone lifts policy success from 11% to 64%.

What they do

Teleoperating a five-fingered hand is slow, and human-hand demonstrations need retargeting and contain no robot actions. DexPort is a UMI-style handle for dexterous hands: the operator carries the actual robot hand (a 20-DoF Wuji hand with a fisheye camera), then bolts the same module onto the robot arm for deployment.

How it works

Hardware. A Meta Quest 2 controller tracks the 6-DoF wrist. The hand-camera module weighs 0.955 kg, and the controller sits above the hand so the grip is near the center of mass.
Skills, not joints. Joystick and buttons drive a few parameterized skills: thumb placement, incremental power grasp, pinch, two-finger grasp, and an RL-trained in-hand rotation (PPO in Isaac Lab) for screwdrivers.
Learning. A diffusion policy with a DINOv2 encoder predicts wrist poses and raw 20-DoF joint targets, with no skill labels. Over 2,500 demonstrations were collected, many in the wild without a robot.

Results

Logging the commanded targets instead of measured joint states raises average task success from 11% to 64%. Measured states under-grip, so objects slip.
Adding 1,000 in-the-wild demonstrations raises pick-and-place success from 47% to 73% on seen objects, 20% to 73% on unseen objects and 40% to 87% with distractors.
122 demonstrations per 30 minutes, 1.4x glove teleoperation. Bare human hands are still 2.3x faster.

Limitations

The device is heavy for long sessions and rotation-heavy motions, and the skill set covers only part of what dexterous hands can do.

Videos

Bimanual barcode scanning, autonomous.

Pick-and-place, autonomous, 1x speed.

Figures

Fig. 1: Transferable Hardware Interface and Control Mapping. (a) Transferable Module: The Wuji hand (A) and fisheye c... — Fig. 1: Transferable Hardware Interface and Control Mapping. (a) Transferable Module: The Wuji hand (A) and fisheye camera (B) form a transferable hand–camera module that mounts to the Quest 2 controller for data collection and to the robot wrist for deployment. Reusing the same module preserves hand embodiment and camera-to-hand extrinsics. (b) Ergonomic Redesign: The final hardware interface places the operator’s grasp near the assembly’s center of mass and adds a support strap, shortening the moment arm and reducing wrist torque. (c) Skill Control Mapping: Quest 2 inputs start and stop demonstration episodes, select hand or screw mode, and modulate thumb, grasp, and screwing skills. The joystick adjusts thumb flexion and abduction in Thumb Mode and incrementally tightens or loosens the grasp in Finger Mode.

Fig. 3: Task Rollouts. We show successful rollouts annotated with the phases of each task. We use these phases to sco... — Fig. 3: Task Rollouts. We show successful rollouts annotated with the phases of each task. We use these phases to score partial success. We also annotate grasp types that emerge in the rollouts, such as a pinch grasp for the cherry tomato and a power grasp for the cucumber. These annotations describe observed behavior. The policy predicts raw joint targets without skill labels.

Fig. 5: In-the-Wild Co-Training. One benefit of DexPort ’s design is its portability. For our pick-and-place task, we... — Fig. 5: In-the-Wild Co-Training. One benefit of DexPort ’s design is its portability. For our pick-and-place task, we collect 1,000 in-the-wild demonstrations (visualized in Fig. 2 ) and co-train with the 700 laboratory demonstrations. We find that increasing the number of cotrained trajectories increases the success rate of the policy in every condition. In the distractor setting, full success rises from 40% to 87%.

Abstract

Collecting high-quality demonstrations remains a central bottleneck for learning dexterous manipulation. Teleoperation records executable robot actions, but it requires a complete robot setup and places the burden of coordinating many finger joints on the operator. Human demonstrations are easier to collect, but their motions must be estimated and retargeted to the robot, and they do not directly provide robot actions. In this paper, we introduce DexPort, a portable data-collection interface for high-DoF robot hands. It uses the same hand hardware for collection and deployment, avoiding a hand embodiment gap, and records both commanded joint targets and measured joint states. Our key observation is that a small set of skills can support a wide range of dexterous tasks. During collection, the operator moves the hand-held interface to specify the robot wrist motion and selects or adjusts skills that coordinate the finger motion. This removes the need to control each joint independently while retaining behaviors such as precise grasping and in-hand manipulation. We collect demonstrations across several real-world tasks and in unstructured environments without a robot arm. We then use behavior cloning to train visuomotor policies directly on the recorded wrist motions and hand commands, without skill labels. We evaluate across 4 tasks: pick-and-place with diverse objects, screwing with in-hand tool use, drawer stowage which requires simultaneous grasp and pulling, and a bimanual scanning task. We collect over 2,500 demonstrations with our portable device, in both laboratory and in-the-wild settings. Across our tasks, we see an improvement from 11% to 64% when using recorded robot commands instead of states, and an improvement from 36% to 78% for task success when using in-the-wild data.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong