Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control

Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, Angjoo Kanazawa

★ Must read April 2021

TL;DR AMP replaces hand-designed motion-tracking rewards with a discriminator trained on unstructured mocap clips, whose score is added as a style reward to a simple task reward in RL. With a 56-clip locomotion dataset, a simulated humanoid picks walking, jogging or running gaits on its own and reaches 0.90 normalized return on target heading, versus 0.46 and 0.63 with walk-only or run-only data.

What they do

Tracking-based character control needs a hand-designed pose-error reward and, for large datasets, a motion planner to pick which clip to track. AMP replaces both with an adversarial motion prior: a discriminator trained on unstructured motion clips whose score becomes a style reward, added to a simple task reward (reach a target, dribble a ball, punch) for RL in simulation.

How it works

  • Reward. Task and style rewards weighted 0.5 each. The discriminator sees state transitions (s, s′) rather than state-action pairs, since mocap has no actions, and gets no phase variable or task features, so the policy is never synchronized to a specific clip.
  • Stable adversarial training. A least-squares discriminator loss with the reward bounded to [0, 1], a gradient penalty on dataset samples (weight 10), and a replay buffer of past policy trajectories.
  • Training. PPO with PD-target actions at 30 Hz in Bullet, on a 34-DoF humanoid, a 59-DoF T-Rex and a 64-DoF dog. Each policy takes 100-300 million samples, about 30-140 h on 16 CPU cores.

Results

  • Target Heading with a 434 s locomotion dataset: the humanoid walks at ~1 m/s, jogs at ~2.5 m/s and runs near 4.5 m/s without being told which gait to use.
  • Skills compose without a planner: walking to a target then punching it (return 0.73), dribbling (0.78), and leaping gaps and rolling under overhead obstacles.
  • Single-clip imitation: pose error stays close to the motion-tracking baseline of Peng et al. (2018a) on most skills (walk 0.030 vs 0.018 m, cartwheel 0.067 vs 0.039 m) and is lower on side-flip (0.124 vs 0.191 m).
  • Ablations: the gradient penalty matters most; without velocity features the character holds a fixed pose on the ground instead of rolling.

Limitations

AMP is prone to mode collapse: with a large, diverse dataset the policy may imitate only a subset of the behaviors. It learns more slowly than tracking with higher pose error, and can get stuck in local optima (on the front-flip the character shuffles forward instead of flipping). The motion prior is retrained from scratch for each policy, the experiments mostly test temporal rather than spatial composition of skills, and all results are in simulation.

Figures

From Figure 1: a simulated character traverses an obstacle course using a variety of locomotion skills.
From Figure 1: a simulated character traverses an obstacle course using a variety of locomotion skills.
From Figure 1: a character learns to walk to a target and punch it.
From Figure 1: a character learns to walk to a target and punch it.
From Figure 2: system overview. A motion prior trained on the motion dataset gives style rewards, which are combined ...
From Figure 2: system overview. A motion prior trained on the motion dataset gives style rewards, which are combined with task rewards to train a policy that satisfies the task goal while moving like the reference motions.
From Figure 4: target speed versus average speed reached on Target Heading by policies trained with the Locomotion, W...
From Figure 4: target speed versus average speed reached on Target Heading by policies trained with the Locomotion, Walk or Run datasets. The Locomotion policy follows the target speeds more closely by imitating different gaits.
Abstract

Synthesizing graceful and life-like behaviors for physically simulated characters has been a fundamental challenge in computer animation. Data-driven methods that leverage motion tracking are a prominent class of techniques for producing high fidelity motions for a wide range of behaviors. However, the effectiveness of these tracking-based methods often hinges on carefully designed objective functions, and when applied to large and diverse motion datasets, these methods require significant additional machinery to select the appropriate motion for the character to track in a given scenario. In this work, we propose to obviate the need to manually design imitation objectives and mechanisms for motion selection by utilizing a fully automated approach based on adversarial imitation learning. High-level task objectives that the character should perform can be specified by relatively simple reward functions, while the low-level style of the character's behaviors can be specified by a dataset of unstructured motion clips, without any explicit clip selection or sequencing. These motion clips are used to train an adversarial motion prior, which specifies style-rewards for training the character through reinforcement learning (RL). The adversarial RL procedure automatically selects which motion to perform, dynamically interpolating and generalizing from the dataset. Our system produces high-quality motions that are comparable to those achieved by state-of-the-art tracking-based techniques, while also being able to easily accommodate large datasets of unstructured motion clips. Composition of disparate skills emerges automatically from the motion prior, without requiring a high-level motion planner or other task-specific annotations of the motion clips. We demonstrate the effectiveness of our framework on a diverse cast of complex simulated characters and a challenging suite of motor control tasks.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.