Emergence of Locomotion Behaviours in Rich Environments

Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, David Silver

Seminal July 2017

TL;DR They train simulated walker, quadruped and humanoid bodies with a distributed PPO on procedurally generated obstacle courses, rewarding mostly forward velocity. Running, jumping, crouching and turning emerge without behaviour-specific reward shaping, and the Planar Walker learns to clear hurdles nearly as tall as its own body.

What they do

Locomotion rewards are usually hand-shaped and brittle. This paper asks whether a rich environment can replace the shaping: train simulated bodies on varied obstacle courses with a reward that is mostly forward velocity, and see which skills appear. To make this scale they introduce Distributed PPO (DPPO).

How it works

DPPO. Workers collect data and compute gradients, which are averaged and applied synchronously on a parameter server. An adaptive KL penalty bounds each update; K-step returns with truncated backprop through time support recurrent policies.
Bodies and courses. Planar Walker (9 DoF, 6 actuated), Quadruped (12 DoF, 8 actuated) and Humanoid (28 DoF, 21 actuated) in MuJoCo. Every episode generates a new course of hurdles, gaps, platforms, slalom walls or variable terrain, with fixed statistics or with difficulty rising along the track.
Reward. Velocity along the track plus a small torque penalty, with a few per-body terms (pose limits, staying near the track center, an alive bonus for the humanoid).
Policy. Separate subnetworks for proprioception and for exteroception (a terrain heightfield sampled around the body); this learned much faster than one fully connected network.

Results

DPPO performs similarly to TRPO, scales with the number of workers, and is faster in wall-clock time than their A3C at equal worker count.
The walker learns to jump hurdles and gaps and crouch under platforms, ending up clearing hurdles nearly as tall as its body. The quadruped jumps obstacles and turns around walls though rewarded only for forward motion.
Hurdle courses that get harder along the track train faster than courses with randomly mixed hurdle heights.
Walker and quadruped policies trained on hurdles are more robust than flat-ground policies to unobserved changes in friction, rubble and the body model, and (walker only) to inclines.

Limitations

Simulation only, with terrain given as a sampled heightfield. The humanoid is much harder: results vary strongly across seeds, are sensitive to algorithm, exploration, reward, termination and initialization, and some runs end in local optima with odd gaits. Humanoid robustness is shown only qualitatively in video, and the algorithm comparison spans different code bases.

Videos

Figures

From Figure 7: Time-lapse sequences of the Humanoid crossing gaps, a hurdle, slalom walls and variable terrain.

From Figure 2: The policy network. One branch processes information local to the body, such as joint and sensor readi... — From Figure 2: The policy network. One branch processes information local to the body, such as joint and sensor readings (blue); a modulating branch processes exteroceptive information such as the terrain shape ahead (green). The output is the mean and standard deviation of the action.

From Figure 3: Terrain types used in training. Left to right and top to bottom: hurdles, platforms, gaps, slalom wall... — From Figure 3: Terrain types used in training. Left to right and top to bottom: hurdles, platforms, gaps, slalom walls, variable terrain.

From Figure 6: (a) Hurdle courses whose height grows along the track (curriculum, green) train faster than courses wi... — From Figure 6: (a) Hurdle courses whose height grows along the track (curriculum, green) train faster than courses with randomly interleaved high and low hurdles (blue), on both easy and hard test courses. (b) Planar Walker (left) and Quadruped (right) policies trained on hurdles (green) are more robust than ones trained on flat terrain (blue) to unobserved changes in friction, rubble, actuator strength and incline. Returns are normalized per setting.

Abstract

The reinforcement learning paradigm allows, in principle, for complex behaviours to be learned directly from simple reward signals. In practice, however, it is common to carefully hand-design the reward function to encourage a particular solution, or to derive it from demonstration data. In this paper explore how a rich environment can help to promote the learning of complex behavior. Specifically, we train agents in diverse environmental contexts, and find that this encourages the emergence of robust behaviours that perform well across a suite of tasks. We demonstrate this principle for locomotion -- behaviours that are known for their sensitivity to the choice of reward. We train several simulated bodies on a diverse set of challenging terrains and obstacles, using a simple reward function based on forward progress. Using a novel scalable variant of policy gradient reinforcement learning, our agents learn to run, jump, crouch and turn as required by the environment without explicit reward-based guidance. A visual depiction of highlights of the learned behavior can be viewed following https://youtu.be/hx_bgoTF7bs .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong