Real-World Humanoid Locomotion with Reinforcement Learning

Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, Koushil Sreenath

★ Must read March 2023

A yoga ball is thrown at the robot and it stays on its feet.

TL;DR A causal transformer that reads the last 16 steps of proprioception and actions is trained with RL in simulation and deployed zero-shot on the full-sized Digit humanoid. Over one week of full-day outdoor testing it never fell, and it recovers from tripping on steps it never saw in training.

What they do

A fully learned walking controller for Digit, a 1.6 m, 45 kg humanoid from Agility Robotics. The policy is a causal transformer over the recent history of proprioceptive observations and actions, trained with RL in simulation and deployed to hardware zero-shot. The hypothesis is that this history lets the model adapt its behavior in context, without updating its weights.

How it works

Two-step training. A teacher MLP sees the full simulator state. The student transformer sees only observations and is trained with PPO plus a KL term toward the teacher, annealed to zero by mid-training so the student can surpass it.
Model. One token per observation-action pair, context window 16, 1.4M parameters. It outputs PD setpoints for 16 joints and PD gains for the 8 actuated leg joints, at 50 Hz over a 1 kHz PD loop.
Simulation. Isaac Gym cannot model Digit’s closed kinematic chains, so stiff “virtual springs” stand in for the rods. Dynamics, control parameters, terrain (smooth and rough planes, slopes), observation noise and delay are randomized; policies are screened in Agility’s simulator before hardware.

Results

One week of full-day outdoor testing on concrete, rubber and grass, wet and dry, without a safety gantry and with no falls.
Indoors it withstands a thrown yoga ball, stick pushes and pulls, floors covered with cables or bubble wrap, slopes up to 8.7% grade and five payloads. In Agility’s simulator it beats the company controller on steps and unstable planks; both do well on slopes.
It reaches a 1 m/s command from rest within 1 s. Arm swing emerges without arm reward terms or reference trajectories, and after hitting an unseen step it lifts its legs higher on later attempts.
Ablations: the transformer beats MLP, TCN and LSTM students; longer context and the joint imitation+RL objective both help.

Limitations

The gait is slightly asymmetric (lateral walking is better to the left), velocity tracking is imperfect, and a very strong pull can still make it fall. The robot is blind, so it bumps into steps and obstacles before reacting.

Videos

Walking across a wooden footbridge in a park.

Walking along a running track.

Figures

From Figure 1: The controller deployed zero-shot in everyday outdoor environments, including plazas, sidewalks, track... — From Figure 1: The controller deployed zero-shot in everyday outdoor environments, including plazas, sidewalks, tracks and grass fields.

From Figure 7: (A) A teacher state policy is trained with full state, then a student observation policy is trained wi... — From Figure 7: (A) A teacher state policy is trained with full state, then a student observation policy is trained with teacher imitation plus RL. (B) Training runs in Isaac Gym across thousands of randomized environments on four A100 GPUs; policies are validated in the manufacturer's high-fidelity simulator before transfer to the real robot. (C) The controller is a causal transformer that predicts the next action from the history of observations and actions.

From Figure 2: Robustness to (A) external disturbances, (B) different terrains and (C) payloads. (D) In simulation th... — From Figure 2: Robustness to (A) external disturbances, (B) different terrains and (C) payloads. (D) In simulation the controller outperforms the company controller, with larger gains on steps and unstable ground.

From Figure 8: (A) The transformer outperforms MLP, TCN and LSTM students. (B) The controller benefits from longer co... — From Figure 8: (A) The transformer outperforms MLP, TCN and LSTM students. (B) The controller benefits from longer context. (C) The joint imitation and RL objective outperforms either term alone.

Abstract

Humanoid robots that can autonomously operate in diverse environments have the potential to help address labour shortages in factories, assist elderly at homes, and colonize new planets. While classical controllers for humanoid robots have shown impressive results in a number of settings, they are challenging to generalize and adapt to new environments. Here, we present a fully learning-based approach for real-world humanoid locomotion. Our controller is a causal transformer that takes the history of proprioceptive observations and actions as input and predicts the next action. We hypothesize that the observation-action history contains useful information about the world that a powerful transformer model can use to adapt its behavior in-context, without updating its weights. We train our model with large-scale model-free reinforcement learning on an ensemble of randomized environments in simulation and deploy it to the real world zero-shot. Our controller can walk over various outdoor terrains, is robust to external disturbances, and can adapt in context.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong