Learning Dexterous In-Hand Manipulation

OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafał Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, Wojciech Zaremba

★ Must read August 2018

TL;DR An LSTM policy trained with RL entirely in heavily randomized simulation reorients a block in a real Shadow Dexterous Hand, sensing the block with motion capture or three cameras. It reaches a median of 13 consecutive goal orientations on the real hand (11.5 with vision), versus 0 for the same policy trained without randomization.

What they do

They train a Shadow Dexterous Hand (24 DoF, tendon-driven) to reorient an object in its palm to a sequence of goal orientations, using reinforcement learning only in simulation and no human demonstrations. Domain randomization, a recurrent policy and large-scale training let it transfer to the real hand, where human-like grasps and finger gaiting emerge without being rewarded.

How it works

Policy. An LSTM trained with PPO sees only fingertip positions (from motion capture) and the object pose, and outputs discretized relative joint targets at about 12 Hz. The value network also sees full simulator state (asymmetric actor-critic).
Randomization. The MuJoCo model is calibrated, then randomized every episode: friction, masses, damping, actuator gains and gravity, plus observation noise, action noise and delay, backlash, marker dropouts and random forces on the object.
Scale. OpenAI Five’s distributed PPO: 384 16-core machines roll out, 8 GPUs optimize, about 2 years of simulated experience per hour.
Vision. A CNN over three RGB cameras predicts object position and orientation, trained only on Unity renders with randomized appearance.

Results

Block, 10 real trials per policy, consecutive goals before failure: median 13 with motion capture, 28.5 with the wrist locked, 11.5 with vision (simulation: 50). Octagonal prism: median 5.
Holding out randomizations on the real hand: median 0 with none, 2 without physics randomization, 2 without unmodeled effects. With vision, dropping observation noise cuts the median from 11.5 to 3.5.
Feed-forward policies reach medians of 3 and 3.5, against 13 for the LSTM.
Full randomization needs about 100 years of simulated experience (around 50 h) to match what 3 years (1.5 h) achieves without it.

Limitations

A reality gap remains: real-hand performance is well below simulation. Each object gets its own policy (the prism policy is finetuned from the block one), and a sphere failed beyond a few rotations. Fingertips still need motion capture. The most common failure is dropping the object while the wrist pitches down, and hardware breakage was a key challenge.

Videos

Figures

From Figure 1: A five-fingered hand trained with reinforcement learning turns a block from an initial to a goal orien... — From Figure 1: A five-fingered hand trained with reinforcement learning turns a block from an initial to a goal orientation, sensing it with vision.

From Figure 2: System overview. (a) Randomized simulations supply data for (b) a recurrent control policy trained wit... — From Figure 2: System overview. (a) Randomized simulations supply data for (b) a recurrent control policy trained with RL and (c) a CNN pose estimator trained separately on rendered images. (d) On the real robot, the CNN reads the object pose from 3 cameras, motion capture gives fingertip positions, and the policy outputs actions.

From Figure 8: Training performance in simulation with groups of randomizations held out. Easier environments, such a... — From Figure 8: Training performance in simulation with groups of randomizations held out. Easier environments, such as no randomizations and no unmodeled effects, train faster.

Abstract

We use reinforcement learning (RL) to learn dexterous in-hand manipulation policies which can perform vision-based object reorientation on a physical Shadow Dexterous Hand. The training is performed in a simulated environment in which we randomize many of the physical properties of the system like friction coefficients and an object's appearance. Our policies transfer to the physical robot despite being trained entirely in simulation. Our method does not rely on any human demonstrations, but many behaviors found in human manipulation emerge naturally, including finger gaiting, multi-finger coordination, and the controlled use of gravity. Our results were obtained using the same distributed RL system that was used to train OpenAI Five. We also include a video of our results: https://youtu.be/jwSbzNHGflM

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong