Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition

Jonah Siekmann, Yesh Godse, Alan Fern, Jonathan Hurst

Seminal November 2020

TL;DR Writing gaits as periodic costs on foot forces and foot speeds gives one reward function, with no reference motions, that trains walking, running, galloping, hopping and skipping on the Cassie biped. The policies transfer from simulation to hardware, and a single policy moves between all two-beat gaits and standing.

What they do

Reference trajectories are hard to obtain and over-constrain a gait; reference-free rewards like “move forward” are underspecified or need gait-specific shaping. This paper uses the fact that every common bipedal gait is a periodic sequence of swing and stance phases per foot: penalize foot force in swing and foot speed in stance, then shift the two feet’s sequences against each other to get different gaits.

How it works

Periodic reward composition. A cycle time φ repeats over [0, 1]. Each reward term multiplies a coefficient, a phase indicator and a measurement (foot force or foot speed). Phase start and end times are Von Mises random variables, and RL optimizes the expected reward, which smooths it at phase boundaries.
Gait parameters. A swing ratio and per-foot cycle offsets define hopping (equal offsets), walking and running (offsets half a cycle apart) and galloping (in between). Skipping uses four phases per foot.
Policy and training. An LSTM sees state estimates, speed commands, clock inputs and phase ratios, and at 40 Hz outputs 10 joint position targets plus PD gains. It is trained with recurrent PPO, auxiliary costs (speed, orientation, torque, smoothness) and dynamics randomization in a MuJoCo Cassie simulator.

Results

Hopping, walking, running and skipping each train as a standalone gait by holding the gait parameters fixed. Each policy uses 150 million samples, which takes 24-36 hours.
In simulation, measured ground reaction forces follow the expected force coefficient for hopping and walking.
With extra transition penalties (keep the feet together when hopping is commanded, stay still when standing is), one policy transitions between all two-beat gaits and standing on hardware.
Outdoors and without perception, that policy hops on and off a sidewalk, walks with one foot on a curb, runs over small bumps and runs down 5 steps of stairs.

Limitations

Varying both offsets and phase ratios during training produced asymmetric walking or blended gaits until behavior-specific transition penalties were added. Skipping is not part of the multi-gait policy. Hardware results are shown only in a video, with no success rates. Quadrupeds, other morphologies and aperiodic motions are left open.

Videos

Figures

From Figure 1: The reward is conditioned on gait parameters, which are also fed to an LSTM policy that outputs PD joi... — From Figure 1: The reward is conditioned on gait parameters, which are also fed to an LSTM policy that outputs PD joint position targets and PD gains to Cassie.

From Figure 3: Swing and stance as two intervals on a circular cycle (top), and the expected foot-force coefficient o... — From Figure 3: Swing and stance as two intervals on a circular cycle (top), and the expected foot-force coefficient over one cycle: -1 in swing, where foot force is penalized, and 0 in stance (bottom).

From Figure 4: Expected foot-force coefficient for each foot in hopping (feet in phase), running (feet half a cycle a... — From Figure 4: Expected foot-force coefficient for each foot in hopping (feet in phase), running (feet half a cycle apart) and skipping (four phases per foot, half a cycle apart).

From Figure 5: Mean simulated ground reaction forces (newtons) for hopping and walking policies against the expected ... — From Figure 5: Mean simulated ground reaction forces (newtons) for hopping and walking policies against the expected foot-force coefficient: the feet push when it is near 0 and stay unloaded near -1.

Abstract

We study the problem of realizing the full spectrum of bipedal locomotion on a real robot with sim-to-real reinforcement learning (RL). A key challenge of learning legged locomotion is describing different gaits, via reward functions, in a way that is intuitive for the designer and specific enough to reliably learn the gait across different initial random seeds or hyperparameters. A common approach is to use reference motions (e.g. trajectories of joint positions) to guide learning. However, finding high-quality reference motions can be difficult and the trajectories themselves narrowly constrain the space of learned motion. At the other extreme, reference-free reward functions are often underspecified (e.g. move forward) leading to massive variance in policy behavior, or are the product of significant reward-shaping via trial-and-error, making them exclusive to specific gaits. In this work, we propose a reward-specification framework based on composing simple probabilistic periodic costs on basic forces and velocities. We instantiate this framework to define a parametric reward function with intuitive settings for all common bipedal gaits - standing, walking, hopping, running, and skipping. Using this function we demonstrate successful sim-to-real transfer of the learned gaits to the bipedal robot Cassie, as well as a generic policy that can transition between all of the two-beat gaits.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong