Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control

Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, Koushil Sreenath

Seminal January 2024

TL;DR One RL recipe trains walking, running and jumping controllers for the Cassie biped, using a policy that reads both a 2-second and a 4-step history of the robot's inputs and outputs, trained across many commanded tasks. Transferred zero-shot, Cassie ran 400 m in 2 min 34 s, 100 m in 27.06 s, and jumped 1.4 m forward and onto a 0.44 m platform.

What they do

A single RL framework that produces walking, running and jumping controllers for Cassie, a human-sized torque-controlled biped. Each policy covers a range of commands for its skill and can also stand. The paper credits the policies’ adaptivity to the robot’s input/output history and their robustness to training on many tasks.

How it works

Dual-history policy. A 1D CNN encodes 2 s (66 steps) of observations and actions. A base MLP gets that embedding plus the last 4 steps directly, a preview of the reference motion and the command, and outputs motor position targets at 33 Hz. Trained end to end with PPO in MuJoCo, with no teacher-student stage.
Reference motions: a trajectory-optimized gait library for walking, retargeted human mocap for running, a hand-made animation for jumping.
Three stages: one fixed task, then randomized commands, then dynamics randomization (friction, link masses, spring stiffness, PD gains, sensor noise, up to 25 ms delay).

Results

In simulation the dual-history policy learns better than residual, state-only, long-only, short-only and RMA/A-RMA variants on all three skills. On hardware it walks in place with the least drift.
The encoder’s latent tracks foot impact forces, take-off and landing, and shifts when dynamics change while tracking error barely moves.
Task randomization adds robustness: pushed while standing, the versatile walking policy steps to recover, while standing-only policies fall.
Hardware: walking command errors of 0.10 m/s, 0.10 m/s and 0.06 m (height), with the same policy still working 325 and 492 days later. Running covered 400 m in 2 min 34 s, 100 m in 27.06 s (peak estimated 4.2 m/s) and slopes up to 10°. Two jumping policies did 19 distinct jumps.

Limitations

Each skill is a separate policy, and adding more skills to one policy risks catastrophic forgetting; after large jumps the robot sometimes oscillates while standing. Jumping onto platforms while turning was too hard to train. The velocity estimator underestimates speed above 3 m/s, and there is no vision.

Videos

Figures

From Figure 4: The multi-stage training framework. Each skill starts from its own reference motion (trajectory optimi... — From Figure 4: The multi-stage training framework. Each skill starts from its own reference motion (trajectory optimization for walking, human mocap for running, animation for jumping), is trained on a single fixed task, then on randomized tasks, then with dynamics randomization, and is transferred zero-shot to Cassie.

From Figure 3: The dual-history controller. A 1D CNN encodes a 2-second history of the robot's inputs and outputs; a ... — From Figure 3: The dual-history controller. A 1D CNN encodes a 2-second history of the robot's inputs and outputs; a base MLP takes that embedding, a 4-step short history, the reference motion and the command, and outputs desired motor positions at 33 Hz. These are low-pass filtered and tracked by joint PD controllers at 2 kHz.

From Figure 6: Learning curves in the hardest training stage (randomized tasks and dynamics) for walking, running and... — From Figure 6: Learning curves in the hardest training stage (randomized tasks and dynamics) for walking, running and jumping, averaged over 3 seeds. The dual-history policy comes close to the privileged expert and beats the residual, state-feedback-only, long-history-only, short-history-only and RMA baselines; RMA fails to learn running.

From Figure 25: Real jumps with the discrete-terrain policy: (i) 0.88 m forward onto a 0.44 m platform, (ii) a 1.4 m ... — From Figure 25: Real jumps with the discrete-terrain policy: (i) 0.88 m forward onto a 0.44 m platform, (ii) a 1.4 m forward jump with a 0.58 s flight phase, (iii) a 1 m forward jump.

Abstract

This paper presents a comprehensive study on using deep reinforcement learning (RL) to create dynamic locomotion controllers for bipedal robots. Going beyond focusing on a single locomotion skill, we develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. Our RL-based controller incorporates a novel dual-history architecture, utilizing both a long-term and short-term input/output (I/O) history of the robot. This control architecture, when trained through the proposed end-to-end RL approach, consistently outperforms other methods across a diverse range of skills in both simulation and the real world. The study also delves into the adaptivity and robustness introduced by the proposed RL system in developing locomotion controllers. We demonstrate that the proposed architecture can adapt to both time-invariant dynamics shifts and time-variant changes, such as contact events, by effectively using the robot's I/O history. Additionally, we identify task randomization as another key source of robustness, fostering better task generalization and compliance to disturbances. The resulting control policies can be successfully deployed on Cassie, a torque-controlled human-sized bipedal robot. This work pushes the limits of agility for bipedal robots through extensive real-world experiments. We demonstrate a diverse range of locomotion skills, including: robust standing, versatile walking, fast running with a demonstration of a 400-meter dash, and a diverse set of jumping skills, such as standing long jumps and high jumps.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong