Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior

Gabriel B. Margolis, Pulkit Agrawal

Seminal December 2022

TL;DR Train one quadruped policy on flat ground only that can walk in many styles (gait, step frequency, body height and pitch, stance width, footswing height), so a human operator can retune it on the spot for stairs, curbs, a 22 cm bar or shoves instead of retraining. On unseen simulated platform terrain, trotting survives 0.88 of the episode vs 0.83 for a gait-free baseline.

What they do

Locomotion policies trained in simulation handle environments like their training set, but when they fail out of distribution the usual fix is to redesign rewards or environments and retrain. Walk These Ways instead trains one quadruped policy, on flat ground only, that can solve the same velocity-tracking task in many different ways (Multiplicity of Behavior, MoB). A human operator then picks the behavior in real time for the situation at hand.

How it works

Behavior parameters. Besides the velocity command, the policy takes an 8-D behavior vector: timing offsets between feet (pronking, trotting, bounding, pacing and their interpolations), stepping frequency, body height, body pitch, stance width and footswing height.
Rewards. Velocity-tracking task rewards, fixed auxiliary rewards for stability and smoothness, and augmented auxiliary rewards for matching the behavior vector. Stance width is tracked through the Raibert heuristic so it does not conflict with fast turning, and the total reward r_task·exp(0.02·r_aux) always pays for task progress.
Training and deployment. PPO in Isaac Gym, with command and behavior resampled within each episode so the policy learns transitions. An MLP sees a 30-step history of joint states, gravity vector, commands and actions. Domain randomization, an actuator network and a modeled 20 ms delay support zero-shot transfer to a Unitree Go1 running at 50 Hz.

Results

Real robot: all gait parameters are tracked after transfer. A high footswing climbs curbs and stairs without tripping, a lowered body crawls under a 22 cm bar (9 cm clearance), a wide stance braces against shoves, and switching from a fast trot to pronking produces a 60 cm leap.
Simulated platform terrain (up to 16 cm, never seen in training): survival fraction 0.88 for trotting and 0.87 for pacing vs 0.83 for the gait-free baseline.
Simulated power consumption, not a training objective: trotting uses 24 vs the baseline’s 35 at 1 m/s.

Limitations

MoB costs in-distribution performance: on flat ground the gait-free baseline tracks velocity better (xy reward 0.94 vs 0.92 for trotting), mostly for combined high linear and angular velocity. A human pilot must choose the behavior; automatic selection is left to future work. The real-world terrain results are demonstrations without reported success rates.

Videos

Figures

From Figure 1: One quadruped policy trained on flat ground, tuned by a human for unseen settings. Top: a low-frequenc... — From Figure 1: One quadruped policy trained on flat ground, tuned by a human for unseen settings. Top: a low-frequency gait slips when sprinting on slippery terrain while a high-frequency one succeeds; low frequency with high footswing climbs stairs; low footswing with a wide stance resists leg shoves. Bottom: other behaviors, including climbing curbs, payload manipulation, crouching under obstacles and rhythmic dance.

From Figure 2: The learned controller switching in place between trotting, pronking, pacing and bounding at alternati... — From Figure 2: The learned controller switching in place between trotting, pronking, pacing and bounding at alternating 2 Hz and 4 Hz, with stance feet highlighted in red.

From Figure 3: Quick behavior transitions at high speed produce an agile leap, emulating a gap crossing on flat groun... — From Figure 3: Quick behavior transitions at high speed produce an agile leap, emulating a gap crossing on flat ground wider than the robot's body length.

Abstract

Learned locomotion policies can rapidly adapt to diverse environments similar to those experienced during training but lack a mechanism for fast tuning when they fail in an out-of-distribution test environment. This necessitates a slow and iterative cycle of reward and environment redesign to achieve good performance on a new task. As an alternative, we propose learning a single policy that encodes a structured family of locomotion strategies that solve training tasks in different ways, resulting in Multiplicity of Behavior (MoB). Different strategies generalize differently and can be chosen in real-time for new tasks or environments, bypassing the need for time-consuming retraining. We release a fast, robust open-source MoB locomotion controller, Walk These Ways, that can execute diverse gaits with variable footswing, posture, and speed, unlocking diverse downstream tasks: crouching, hopping, high-speed running, stair traversal, bracing against shoves, rhythmic dance, and more. Video and code release: https://gmargo11.github.io/walk-these-ways/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong