DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills

Xue Bin Peng, Pieter Abbeel, Sergey Levine, Michiel van de Panne

★ Must read April 2018

TL;DR DeepMimic trains physically simulated characters with RL to imitate motion clips, rewarding closeness to the reference pose plus an optional task goal. One recipe learns 24 humanoid skills, from walking to backflips, and adding a task reward raises target-strike success from 19% (imitation only) to 99%.

What they do

Deep RL controllers for simulated characters tend to move unnaturally. DeepMimic uses PPO to train a policy that reproduces a reference clip (motion capture or keyframed animation) in physics simulation, optionally while pursuing a task such as a target heading. Characters are a humanoid, a simulated Atlas, a T-Rex and a dragon.

How it works

Policy. Inputs are link positions, rotations and velocities, a phase variable for progress through the clip, and an optional goal. Outputs are PD target angles at 30 Hz.
Reward. 0.7 × imitation (match the reference joint orientations, joint velocities, hand and foot positions and center of mass) + 0.3 × task.
Reference state initialization (RSI) starts each episode at a random state from the clip, so the policy sees mid-flip states before it can reach them. Early termination (ET) ends the episode when the torso or head touches the ground.
Multiple clips: a max over per-clip imitation rewards, a one-hot skill selector, or separately trained policies chosen at runtime by their value functions.

Results

24 humanoid skills learned from 0.5-5 s clips, with normalized returns from 0.485 (frontflip) to 0.985 (walk). Atlas policies trained on the retargeted clips reach 0.846 (run) and 0.630 (backflip); humanoid policies run directly on the Atlas get 0.013 and 0.014.
Strike success is 99% with imitation plus task reward vs 19% with imitation only; throwing is 75% vs 5%. Task-only policies succeed more often at throwing (93%) but move unnaturally, running to the target with the ball.
Without ET, backflip return drops from 0.791 to 0.379. Without RSI the return is similar (0.730), but the character only hops backwards instead of flipping.
With no pushes during training, the run policy survives 720 N forward pushes lasting 0.2 s.

Limitations

The phase variable advances linearly with time, so the policy cannot adjust the motion’s timing. Multi-clip methods are shown only for small sets of clips. PD gains and reward weights are hand-set. Each humanoid skill takes about 60 million samples (about 2 days on an 8-core CPU) and is trained separately. All results are in simulation.

Videos

Figures

From Figure 1: skills learned by imitating motion-capture clips. Left, the simulated humanoid does a cartwheel; right... — From Figure 1: skills learned by imitating motion-capture clips. Left, the simulated humanoid does a cartwheel; right, the simulated Atlas robot does a spinkick.

From Figure 2: the visuomotor policy network. A terrain heightmap passes through 3 convolutional layers and a 64-unit... — From Figure 2: the visuomotor policy network. A terrain heightmap passes through 3 convolutional layers and a 64-unit layer, is concatenated with the state and goal, and goes through fully connected layers of 1024 and 512 units to the action mean. Without a heightmap only the fully connected part is used.

From Figure 7: a baseball-pitch policy trained to throw a ball to a target.

From Figure 11: backflip learning curves with both reference state initialization and early termination (RSI + ET), w... — From Figure 11: backflip learning curves with both reference state initialization and early termination (RSI + ET), with ET only and with RSI only.

Abstract

A longstanding goal in character animation is to combine data-driven specification of behavior with a system that can execute a similar behavior in a physical simulation, thus enabling realistic responses to perturbations and environmental variation. We show that well-known reinforcement learning (RL) methods can be adapted to learn robust control policies capable of imitating a broad range of example motion clips, while also learning complex recoveries, adapting to changes in morphology, and accomplishing user-specified goals. Our method handles keyframed motions, highly-dynamic actions such as motion-captured flips and spins, and retargeted motions. By combining a motion-imitation objective with a task objective, we can train characters that react intelligently in interactive settings, e.g., by walking in a desired direction or throwing a ball at a user-specified target. This approach thus combines the convenience and motion quality of using motion clips to define the desired style and appearance, with the flexibility and generality afforded by RL methods and physics-based animation. We further explore a number of methods for integrating multiple clips into the learning process to develop multi-skilled agents capable of performing a rich repertoire of diverse skills. We demonstrate results using multiple characters (human, Atlas robot, bipedal dinosaur, dragon) and a large variety of skills, including locomotion, acrobatics, and martial arts.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong