Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning

Jonah Siekmann, Kevin Green, John Warila, Alan Fern, Jonathan Hurst

Seminal May 2021

TL;DR An LSTM policy trained with PPO in simulation on randomized staircases lets the Cassie biped climb and descend real stairs blind, using only proprioception and the existing flat-ground reward. On an outdoor staircase it succeeded in 8 of 10 ascents and 10 of 10 descents.

What they do

The authors ask how far a blind biped can go on stairs. They take an existing sim-to-real RL framework for flat-ground Cassie gaits and change only the training terrain to randomized staircases, with no stair-specific reward terms. The policy uses only proprioception, with no cameras or terrain model.

How it works

Policy. A two-layer LSTM at 40 Hz takes pelvis orientation and angular velocity, joint states, operator commands and gait-phase clocks. It outputs PD targets for 10 joints (tracked at 2 kHz) plus a stepping-frequency adjustment.
Reward. Unchanged from prior work: foot forces and foot velocities are penalized at alternating gait phases, plus velocity, orientation, smoothness and energy terms. No reference trajectories.
Terrain randomization. Each episode places 1 to 8 steps (rise 10 to 21 cm, run 24 to 30 cm, ±1 cm noise) up to 10 m ahead, with no curriculum. Dynamics are also randomized.
Training. PPO in MuJoCo for 300 million timesteps.

Results

In simulation (150 trials on a five-step staircase), Stair LSTM policies reached the top and bottom most often. Feedforward policies trained on the same stairs did poorly, and Flat Ground LSTM policies almost never climbed.
On hardware, one randomly selected policy climbed 8 of 10 and descended 10 of 10 times on an outdoor staircase. It also handled curbs, logs, wet grass and a small hill, none of which were modeled in training.
Cost of transport at 1 m/s on flat ground was 0.46 for Stair LSTM versus 0.38 for Flat Ground LSTM. A binary “stairs within 1 m” input brought this down to 0.39 when no stairs were near.

Limitations

Success depends on approach speed: policies fail more often when slow or fast. The policy falls after particularly bad missteps. It seems to need a solid stair face: slatted stairs caused many more failures in simulation, even when included in training. Hardware success rates come from 10 trials per direction on a single staircase.

Videos

Figures

From Figure 1: A learned controller maps proprioceptive data to PD targets. It is trained on domain-randomized stairs... — From Figure 1: A learned controller maps proprioceptive data to PD targets. It is trained on domain-randomized stairs in simulation and run on the hardware, where it blindly climbs and descends real stairs.

From Figure 2: Stair parameters randomized at the start of each simulated episode: stair count (1 to 8), rise (0.10 t... — From Figure 2: Stair parameters randomized at the start of each simulated episode: stair count (1 to 8), rise (0.10 to 0.21 m), run (0.24 to 0.30 m), landing width (0.50 to 2.00 m) and ground slope before and after the stairs.

From Figure 4: Probability of climbing (top) and descending (bottom) stairs without falling, against commanded speed ... — From Figure 4: Probability of climbing (top) and descending (bottom) stairs without falling, against commanded speed from 0.25 to 1.5 m/s, over 150 simulated trials. Stair LSTM policies do best, with separate best speeds for ascent and descent. Stair FF policies score low, and Flat Ground LSTM policies almost never climb.

Abstract

Accurate and precise terrain estimation is a difficult problem for robot locomotion in real-world environments. Thus, it is useful to have systems that do not depend on accurate estimation to the point of fragility. In this paper, we explore the limits of such an approach by investigating the problem of traversing stair-like terrain without any external perception or terrain models on a bipedal robot. For such blind bipedal platforms, the problem appears difficult (even for humans) due to the surprise elevation changes. Our main contribution is to show that sim-to-real reinforcement learning (RL) can achieve robust locomotion over stair-like terrain on the bipedal robot Cassie using only proprioceptive feedback. Importantly, this only requires modifying an existing flat-terrain training RL framework to include stair-like terrain randomization, without any changes in reward function. To our knowledge, this is the first controller for a bipedal, human-scale robot capable of reliably traversing a variety of real-world stairs and other stair-like disturbances using only proprioception.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong