A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning

Laura Smith, Ilya Kostrikov, Sergey Levine

Seminal August 2022

TL;DR An A1 quadruped learns to walk from scratch in the real world with model-free RL, using SAC with a regularized critic, many critic updates per step and a fast JAX implementation. It learns a gait in about 20 minutes on each of five indoor and outdoor terrains.

What they do

The paper asks how fast fully model-free deep RL can learn quadruped locomotion directly on hardware. Using no new algorithm, only existing components implemented carefully and a well-chosen task setup, a Unitree A1 learns to walk from scratch in about 20 minutes, on flat ground, a memory foam mattress, mulch, a lawn and a hiking trail.

How it works

Algorithm. SAC with critic regularization in the style of DroQ (layer norm, dropout), taking 20 critic updates and one actor update between every pair of actions. Training is synchronous, per step, after 1000 random-action steps.
Speed. JAX with the 20 critic updates jit-compiled into one call gives 2400 critic updates per second on a laptop RTX 2070. Their best PyTorch version allowed only 7.5 Hz control.
MDP. Proprioceptive state only. Actions are PD targets for 12 joints at 20 Hz, kept within a fixed offset of a default pose. The reward is constant within a target-velocity band, using forward velocity projected onto the ground plane from an onboard Kalman-filter estimator.
Resets. Episodes end when roll or pitch exceed 30 degrees; a learned reset policy from prior work stands the robot up.

Results

On every terrain the robot learns to walk in under 20,000 samples, about 17 minutes of data and 20 minutes of wall-clock time; flat ground was repeated four times.
In their comparison table, prior work that learned joint PD targets from scratch on real robots used 60,000-160,000 samples and 1-2 hours.
In simulation, plain SAC at 20 updates per step needs about 30k samples to reach the target velocity; REDQ saves about 5k, and layer norm alone does better still. An unconstrained action space makes no progress, and low-pass filtering the actions slows learning.

Limitations

The task is forward walking at a target velocity. A person lifts and reorients the robot to keep it in the training area. The onboard velocity estimator is significantly less accurate than VR tracking and systematically underestimates velocity. Real-time training depends on a fast, jit-compiled implementation.

Videos

Figures

From Figure 1: Training locations: flat ground (blue), soft, irregular mulch (green), grass (red) and a hiking trail ... — From Figure 1: Training locations: flat ground (blue), soft, irregular mulch (green), grass (red) and a hiking trail (yellow). The robot acquires a gait on each within 20 minutes of training.

From Figure 3: Learned gaits. Left to right: flat ground covered in dense foam mats, a 5 cm memory foam mattress, loo... — From Figure 3: Learned gaits. Left to right: flat ground covered in dense foam mats, a 5 cm memory foam mattress, loose eucalyptus bark, a grassy lawn and a gently sloped hiking trail.

From Figure 4: Indoor learning curves, average policy velocity against real-world wall-clock time, on flat ground (me... — From Figure 4: Indoor learning curves, average policy velocity against real-world wall-clock time, on flat ground (mean and standard deviation over four runs) and on memory foam.

From Figure 4: Outdoor learning curves, average policy velocity against real-world wall-clock time, on the hiking tra... — From Figure 4: Outdoor learning curves, average policy velocity against real-world wall-clock time, on the hiking trail, lawn and mulch.

Abstract

Deep reinforcement learning is a promising approach to learning policies in uncontrolled environments that do not require domain knowledge. Unfortunately, due to sample inefficiency, deep RL applications have primarily focused on simulated environments. In this work, we demonstrate that the recent advancements in machine learning algorithms and libraries combined with a carefully tuned robot controller lead to learning quadruped locomotion in only 20 minutes in the real world. We evaluate our approach on several indoor and outdoor terrains which are known to be challenging for classical model-based controllers. We observe the robot to be able to learn walking gait consistently on all of these terrains. Finally, we evaluate our design decisions in a simulated environment.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong