DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, C. Karen Liu

Seminal March 2024

Mocap demonstrations with point clouds, then fully autonomous robot rollouts of the trained policies.

TL;DR A wearable rig of mocap gloves, SLAM wrist cameras and a chest RGB-D camera records human hand motion with 3D point clouds, and DexIL retargets it to two LEAP hands and trains a point-cloud diffusion policy. With no robot data, policies reach 72% average success on three tasks, and collection is three times faster than teleoperation.

What they do

Teleoperating dexterous hands is slow and needs the robot; vision-based hand tracking breaks under occlusion and lacks accurate 3D. DexCap is a portable mocap rig that records finger motion, 6-DoF wrist poses and a 3D view of the scene without any robot. DexIL turns those recordings into policies for two Franka arms with LEAP hands, with optional human-in-the-loop correction.

How it works

Capture. Rokoko EMF gloves track the fingertips, a T265 camera on each glove tracks the wrist with SLAM, and a chest L515 RGB-D LiDAR records the scene, its pose tracked by a third T265. Calibration on a 3D-printed rack takes about 10 s. The rig runs 40 minutes on a power bank, weighs 3.96 lb and costs under $4k.
Retargeting. RGB-D frames become point clouds in a fixed world frame, which removes torso motion, and are shifted into the robot workspace. Fingertip IK, initialized from the wrist pose, maps the human hand to the LEAP hand, which is about 50% larger.
Policy. A diffusion policy with a Perceiver point-cloud encoder; robot hand links placed by forward kinematics are merged into the point cloud.
Correction. During rollouts a DexCap wearer adds residual corrections, or switches to direct teleoperation with a foot pedal. Corrections and original data are sampled equally for fine-tuning.

Results

From 30 minutes of mocap and no robot data: 72% average success on sponge picking, ball collecting and plate wiping. Full-image policies get 0%, masked images 42%.
Packaging from one hour of in-the-wild data (over 10 scenes): 47% full task, 40% on unseen objects; 57% after 30 human corrections.
Corrections make hard tasks possible: scissor cutting rises from 0% to 20%, tea preparing from 0% to 25% (uncapping 30% to 65%).
Collection takes 7.75 s per demo against 21.50 s for teleoperation; SLAM wrist drift is 0.8 cm versus 11.3 cm for IMU tracking.

Limitations

Battery life caps sessions at 40 minutes. Fingertip IK cannot fully bridge the size gap between human and robot hands (piano playing is given as an example). The system records no force or touch, which made stabilizing the box in Packaging hard.

Videos

The wearable rig, and hand mocap shown live in a SLAM-stabilized point cloud.

Ball collecting by bare hand, with DexCap and by teleoperation: 5.17, 7.75 and 21.50 s per demo.

Figures

From Figure 1: DexCap collects human hand mocap data and 3D observations in the wild; DexIL adapts it to the robot an... — From Figure 1: DexCap collects human hand mocap data and 3D observations in the wild; DexIL adapts it to the robot and trains a policy for the same task.

From Figure 2: A chest rack holds an L515 LiDAR camera and three T265 tracking cameras; a backpack NUC and power bank... — From Figure 2: A chest rack holds an L515 LiDAR camera and three T265 tracking cameras; a backpack NUC and power bank run it for about 40 minutes; two T265s start in a known pose on the rack, then click onto the gloves to track the palms.

From Figure 4: (a) DexIL turns RGB-D frames into point clouds in robot space, retargets hand motion with fingertip IK... — From Figure 4: (a) DexIL turns RGB-D frames into point clouds in robot space, retargets hand motion with fingertip IK, and learns a policy that outputs future goal positions. (b) Humans add residual actions during rollouts; the corrections are sampled alongside the original data for fine-tuning.

From Figure 8: Data collection throughput on Ball collecting. DexCap is close to natural human motion and three times... — From Figure 8: Data collection throughput on Ball collecting. DexCap is close to natural human motion and three times faster than teleoperation.

Abstract

Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. Despite this potential, substantial challenges persist, particularly with the portability of existing hand motion capture (mocap) systems and the complexity of translating mocap data into effective robotic policies. To tackle these issues, we introduce DexCap, a portable hand motion capture system, alongside DexIL, a novel imitation algorithm for training dexterous robot skills directly from human hand mocap data. DexCap offers precise, occlusion-resistant tracking of wrist and finger motions based on SLAM and electromagnetic field together with 3D observations of the environment. Utilizing this rich dataset, DexIL employs inverse kinematics and point cloud-based imitation learning to seamlessly replicate human actions with robot hands. Beyond direct learning from human motion, DexCap also offers an optional human-in-the-loop correction mechanism during policy rollouts to refine and further improve task performance. Through extensive evaluation across six challenging dexterous manipulation tasks, our approach not only demonstrates superior performance but also showcases the system's capability to effectively learn from in-the-wild mocap data, paving the way for future data collection methods in the pursuit of human-level robot dexterity. More details can be found at https://dex-cap.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong