VideoDex: Learning Dexterity from Internet Videos

Kenneth Shaw, Shikhar Bahl, Deepak Pathak

Seminal December 2022

Pick task, the hand lifts an unseen paper cup, 1x speed.

TL;DR Retarget human hand motion from Epic Kitchens videos into robot hand and arm trajectories, pretrain an open-loop policy on them, then finetune on teleoperated demos. On a LEAP Hand and xArm, VideoDex picks unseen objects with 0.77 success versus at most 0.44 for baselines without the human-video prior.

What they do

Teleoperating a dexterous hand for every task is slow, and most work uses human video only to pretrain visual features. VideoDex also takes the motion from egocentric human videos in Epic Kitchens: it converts each clip’s hand and wrist movement into a trajectory for a 16-DoF LEAP Hand on an xArm 6, pretrains the policy on those trajectories (an action prior), and then trains on 120-175 teleoperated robot demos per task. The policy is open-loop: from the first image it outputs the whole hand and wrist trajectory.

How it works

Finger retargeting. OpenPose crops the hand, FrankMocap fits MANO shape and pose, and a network trained on the keypoint-vector energy from Robotic Telekinesis maps that pose to robot hand joints.
Wrist retargeting. PnP gives the wrist pose in each frame, ORB-SLAM3 compensates for the moving head camera, and gravity comes from the surface normal of tables and counters (Detic segmentation plus Adabins monocular depth). A heuristic rescales and centers the trajectory in the robot workspace.
Policy. An R3M ResNet-18 encodes the first image (visual prior); two Neural Dynamic Policies, one for the wrist and one for the hand, produce smooth trajectories (physical prior). It is pretrained on retargeted clips of the same task category (close to 3000 for pick); training takes about 10 h on one 2080Ti.

Results

Seven real tasks, success on unseen test objects: pick 0.77 vs 0.44 for the best baseline, place 0.80 vs 0.50, open 0.80 vs 0.60, rotate 0.71 vs 0.56. Push and uncover are near ceiling for all methods, and on training objects BC-NDP beats VideoDex on rotate (0.94 vs 0.85) and open (0.90 vs 0.80).
With 5 demos per variant, VideoDex still reaches 30% success on unseen objects.
On place, swapping R3M for ImageNet features drops success from 0.70 to 0.40. Estimating gravity from surface normals beats a fixed, random or IMU-based estimate.
The action prior also improves a 2-finger xArm gripper.

Limitations

Policies are open-loop and cannot react to changes in the scene. Off-the-shelf hand detectors often give wrong 6D poses while the hand touches objects. The retargeting must be recomputed for each robot embodiment, and the method uses curated, action-labelled Epic Kitchens clips rather than raw internet video.

Videos

Open task, the hand pulls out the drawer of an unseen box, 1x speed.

Cover task, the hand drapes a cloth over a plate of fruit, 1x speed.

Figures

From Figure 1: Human videos are retargeted as an action prior, pretrained embeddings serve as a visual prior, and Neu... — From Figure 1: Human videos are retargeted as an action prior, pretrained embeddings serve as a visual prior, and Neural Dynamic Policies as a physical prior, to complete many tasks on a robot hand.

From Figure 4: Detected human fingers are converted to robot fingers with a learned energy function; the wrist is ret... — From Figure 4: Detected human fingers are converted to robot fingers with a learned energy function; the wrist is retargeted from the detections and the camera trajectory and transformed to the robot arm.

From Figure 6: Without any robot training, a network initialized with the human-video action prior (left) predicts tr... — From Figure 6: Without any robot training, a network initialized with the human-video action prior (left) predicts trajectories closer to the ground-truth robot trajectories than one initialized with only a visual prior (right).

Abstract

To build general robotic agents that can operate in many environments, it is often imperative for the robot to collect experience in the real world. However, this is often not feasible due to safety, time, and hardware restrictions. We thus propose leveraging the next best thing as real-world experience: internet videos of humans using their hands. Visual priors, such as visual features, are often learned from videos, but we believe that more information from videos can be utilized as a stronger prior. We build a learning algorithm, VideoDex, that leverages visual, action, and physical priors from human video datasets to guide robot behavior. These actions and physical priors in the neural network dictate the typical human behavior for a particular robot task. We test our approach on a robot arm and dexterous hand-based system and show strong results on various manipulation tasks, outperforming various state-of-the-art methods. Videos at https://video-dex.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong