Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning

Nikita Rudin, David Hoeller, Philipp Reist, Marco Hutter

★ Must read September 2021

TL;DR Run simulation and PPO training for thousands of quadrupeds in parallel on a single GPU, with a game-inspired terrain curriculum. A rough-terrain ANYmal policy trains in under 20 minutes (flat terrain in under 4) and transfers to the real robot.

What they do

In prior work, training quadruped locomotion policies took between 12 and 120 hours. Here, Isaac Gym runs both simulation and PPO on one workstation GPU, with thousands of ANYmal robots stepping at once. The authors study how PPO must be adapted to this massively parallel setting and train a perceptive rough-terrain policy that runs on the real robot.

How it works

  • Parallel PPO. Batch size = robots × steps per robot, so each robot contributes only a few steps per update. Learning struggles below about 25 consecutive steps (0.5 s), because GAE needs rewards from several steps. Mini-batches hold tens of thousands of samples.
  • Time-out bootstrapping. The critic cannot predict time-out resets, so at a time-out its target is bootstrapped with its own value estimate. This raises total reward by about 10-20%.
  • Game-inspired curriculum. One mesh tiles five terrain types (flat, slopes, rough, obstacles, stairs) by difficulty: step height goes from 5 to 20 cm and slope from 0 to 25°. A robot that walks off its tile moves up a level. One that covers less than half its commanded distance moves down. Robots that solve the top level are sent back to a random level.
  • Policy. Inputs are proprioception and 108 terrain heights; outputs are PD joint targets, with no gait-specific terms. Sim-to-real uses friction randomization, observation noise, random pushes and an LSTM actuator network.

Results

  • With 4096 robots and a batch of 98304, 1500 updates take under 20 minutes. Training time scales nearly linearly up to about 4000 robots. With too many robots, performance drops sharply because each robot’s horizon gets too short.
  • In simulation, success is nearly 100% on stairs up to 0.2 m. Beyond 25° the robots can no longer climb slopes.
  • ANYmal B and an arm-equipped ANYmal C train with no changes; Unitree A1 and Cassie need small ones.
  • On hardware, ANYmal C walks up and down stairs and over obstacles using a Lidar elevation map.

Limitations

The imperfect real elevation map forces the hardware speed limit down to 0.6 m/s. Policies often show artifacts, such as a dragging leg, that take reward tuning to remove. The minimum number of steps per robot caps how many robots can run in parallel, so the GPU is not fully used.

Videos

Full video (YouTube)

Figures

From Figure 1: Thousands of robots learning to walk in simulation.
From Figure 1: Thousands of robots learning to walk in simulation.
From Figure 3: 4000 robots spread across the terrains after 1000 policy updates of the automatic curriculum. They sta...
From Figure 3: 4000 robots spread across the terrains after 1000 policy updates of the automatic curriculum. They start on the first row, closest to the camera, and progressively reach harder terrain.
From Figure 4: Total training time against the number of parallel robots for three batch sizes, compared with the 200...
From Figure 4: Total training time against the number of parallel robots for three batch sizes, compared with the 20000-robot baseline (dashed).
From Figure 7: The real ANYmal C on a staircase, running the policy trained in simulation.
From Figure 7: The real ANYmal C on a staircase, running the policy trained in simulation.
Abstract

In this work, we present and study a training set-up that achieves fast policy generation for real-world robotic tasks by using massive parallelism on a single workstation GPU. We analyze and discuss the impact of different training algorithm components in the massively parallel regime on the final policy performance and training times. In addition, we present a novel game-inspired curriculum that is well suited for training with thousands of simulated robots in parallel. We evaluate the approach by training the quadrupedal robot ANYmal to walk on challenging terrain. The parallel approach allows training policies for flat terrain in under four minutes, and in twenty minutes for uneven terrain. This represents a speedup of multiple orders of magnitude compared to previous work. Finally, we transfer the policies to the real robot to validate the approach. We open-source our training code to help accelerate further research in the field of learned legged locomotion.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.