DexMV: Imitation Learning for Dexterous Manipulation from Human Videos

Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, Xiaolong Wang

Seminal August 2021

Human videos, estimated 3D hand-object poses and the retargeted robot hand side by side.

TL;DR DexMV records human hand videos, estimates 3D hand and object poses, and retargets them into demonstrations for a 30-DoF simulated Adroit hand. With these demonstrations DAPG relocates four of five YCB objects at 93-100% success, where RL alone reaches 0-67%.

What they do

Reinforcement learning for a multi-finger hand needs a lot of data, and earlier VR-glove demonstrations are costly: there were only 25 per task. DexMV records ordinary human videos instead, about 100 demonstrations per hour, and turns them into demonstrations for a 30-DoF Adroit hand in MuJoCo. The tasks are relocate (five YCB objects), pour and place inside.

How it works

Capture and pose estimation. Two RealSense D435 cameras record the human. PVN3D estimates 6-DoF object poses. MANO hand poses are estimated per frame, then refined by optimizing against 2D joints and depth from both cameras.
Hand motion retargeting. An optimization maps the 45-DoF MANO hand to the robot’s joint angles. It matches task space vectors from the palm to the fingertips and to the middle phalanges, so finger bending is preserved, and adds a smoothness term.
Action estimation. The joint trajectory is fit as a minimum-jerk function, resampled from the 30 Hz video to the 120 Hz simulation, and inverse dynamics gives the actions.
Imitation learning. The demonstrations augment TRPO through DAPG and GAIL+ (state-action) and SOIL (state-only).

Results

Relocate success: DAPG reaches 1.00 on tomato can, clamp and mug and 0.93 on mustard bottle. RL alone gets 0.06 on mustard bottle and 0.00 on sugar box, where only SOIL succeeds (0.67).
Pour: DAPG pours 27.2% of the particles into the container. Pure RL rarely pours any.
Unseen ShapeNet objects: DAPG relocate success is 68.5-83.6% within the training categories and 33.4-49.9% on cameras, a new category.
Ablations: the phalanx vectors beat fingertip-only retargeting, removing the smoothness term causes a large drop, and more demonstrations help.

Limitations

Every policy is trained and tested only in simulation. Retargeting from 51 human DoF to 30 robot DoF loses information. The recorded videos have no ground-truth poses, so pose estimation was evaluated on DexYCB instead.

Videos

The simulated Adroit hand pours particles from a mug into a container.

The simulated Adroit hand picks up a banana and places it inside a mug.

Figures

From Figure 1: Human videos of manipulation tasks (top), the 3D hand-object poses estimated from them (middle), and t... — From Figure 1: Human videos of manipulation tasks (top), the 3D hand-object poses estimated from them (middle), and the same relocate, pour and place inside tasks for the multi-finger robot in simulation (bottom).

From Figure 2: DexMV platform and pipeline. A computer vision system (yellow) records human videos, 3D hand-object po... — From Figure 2: DexMV platform and pipeline. A computer vision system (yellow) records human videos, 3D hand-object pose estimation and demonstration translation (green) turn them into robot demonstrations, and a simulation system (blue) uses them for imitation learning.

From Figure 3: Top, the human and robot kinematic chains and the ten task space vectors matched between the two hands... — From Figure 3: Top, the human and robot kinematic chains and the ten task space vectors matched between the two hands. Bottom, demonstration translation by hand motion retargeting and then action estimation.

From Figure 5: Learning curves of SOIL, GAIL+, DAPG and RL on relocate for five objects, over training iterations and... — From Figure 5: Learning curves of SOIL, GAIL+, DAPG and RL on relocate for five objects, over training iterations and three random seeds.

Abstract

While significant progress has been made on understanding hand-object interactions in computer vision, it is still very challenging for robots to perform complex dexterous manipulation. In this paper, we propose a new platform and pipeline DexMV (Dexterous Manipulation from Videos) for imitation learning. We design a platform with: (i) a simulation system for complex dexterous manipulation tasks with a multi-finger robot hand and (ii) a computer vision system to record large-scale demonstrations of a human hand conducting the same tasks. In our novel pipeline, we extract 3D hand and object poses from videos, and propose a novel demonstration translation method to convert human motion to robot demonstrations. We then apply and benchmark multiple imitation learning algorithms with the demonstrations. We show that the demonstrations can indeed improve robot learning by a large margin and solve the complex tasks which reinforcement learning alone cannot solve. More details can be found in the project page: https://yzqin.github.io/dexmv

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong