Rapid Locomotion via Reinforcement Learning

Gabriel B. Margolis, Ge Yang, Kartik Paigwar, Tao Chen, Pulkit Agrawal

Seminal May 2022

TL;DR One RL policy trained in simulation, with an adaptive curriculum over velocity commands and teacher-student online system identification, runs the MIT Mini Cheetah at up to 3.9 m/s and spins it at 5.7 rad/s, deployed zero-shot.

What they do

A single neural network controller for the MIT Mini Cheetah, trained with RL in simulation and deployed zero-shot, tracks commanded forward, lateral and yaw velocity up to high speeds. Sampling fast commands uniformly from the start makes training fail, so they add an automatic curriculum over commands and reuse teacher-student online system identification from prior work.

How it works

Policy. Inputs are joint angles and velocities, the IMU gravity vector, the previous action and the velocity command; outputs are joint position targets for a PD controller at 50 Hz. PPO in Isaac Gym: 400M steps from 4000 parallel agents, under three hours on one RTX 3090.
Grid Adaptive curriculum. Forward and yaw commands are sampled jointly from a 0.5 m/s by 0.5 rad/s grid that starts at [-1, 1] for both. When a command’s tracking rewards pass a threshold, its grid neighbors are added. Unlike the Box Adaptive variant, which samples the two independently, it avoids infeasible combinations of fast running and turning.
Online system identification. A teacher sees the randomized friction, restitution, payload, center of mass and motor strength through an encoder. A student module regresses the same latent from 15 steps of history and reuses the teacher’s policy body.

Results

Indoors (motion capture, 6.0 m/s command): 3.81 m/s average across seeds, 3.9 m/s top sustained speed, versus the previous 3.7 m/s record from model-predictive control on this robot. Froude number 5.1. Without system identification: 2.49 m/s.
Outdoors: a 10 m grass dash in 2.94 s (3.4 m/s). Yaw rate reaches 5.7 rad/s, 90% of the 6.28 rad/s model-based record, and the robot stays stable spinning on ice.
Without a curriculum the policy fails to learn; Grid Adaptive covers a larger command area than Box Adaptive at all error thresholds. The student nearly matches the teacher, and both beat domain randomization.
An MPC baseline did not recover from slipping on a gravel incline or tripping over a barrier; the learned policy did.

Limitations

A large sim-to-real gap at top speed remains (5.46 m/s in simulation vs 3.81 m/s real). Outdoor tests are qualitative, without motion capture. The policy only controls planar body velocity and has no vision, so it cannot plan for stairs or pitfalls. It was trained only on flat ground; adding terrain roughness in training lowers high-speed performance.

Videos

Figures

From Figure 1: High-speed spinning indoors at 5.7 rad/s. The same network also sprints at 3.9 m/s, runs a 10 m grass ... — From Figure 1: High-speed spinning indoors at 5.7 rad/s. The same network also sprints at 3.9 m/s, runs a 10 m grass dash at 3.4 m/s and spins on ice.

From Figure 2: The controller is a 5-layer neural network trained in simulation that maps a history of joint encoder,... — From Figure 2: The controller is a 5-layer neural network trained in simulation that maps a history of joint encoder, IMU, past action and velocity command inputs to desired joint positions.

From Figure 3: Command area vs. error threshold. The Grid Adaptive curriculum tracks a larger range of velocities tha... — From Figure 3: Command area vs. error threshold. The Grid Adaptive curriculum tracks a larger range of velocities than Box Adaptive at every threshold; No Curriculum fails to learn velocity control.

From Figure 4: Online system identification reduces tracking error, particularly at high speeds. Command area vs. err... — From Figure 4: Online system identification reduces tracking error, particularly at high speeds. Command area vs. error threshold for the teacher, student and domain-randomized policies.

Abstract

Agile maneuvers such as sprinting and high-speed turning in the wild are challenging for legged robots. We present an end-to-end learned controller that achieves record agility for the MIT Mini Cheetah, sustaining speeds up to 3.9 m/s. This system runs and turns fast on natural terrains like grass, ice, and gravel and responds robustly to disturbances. Our controller is a neural network trained in simulation via reinforcement learning and transferred to the real world. The two key components are (i) an adaptive curriculum on velocity commands and (ii) an online system identification strategy for sim-to-real transfer leveraged from prior work. Videos of the robot's behaviors are available at: https://agility.csail.mit.edu/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong