Learning Quadrupedal Locomotion over Challenging Terrain

Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter

★ Must read October 2020

TL;DR A blind quadruped controller trained in simulation, where an RL teacher with privileged terrain information is distilled into a student that reads 2 s of joint and IMU history. It transfers zero-shot to mud, snow, vegetation and running water, and ran four 60-minute DARPA SubT missions with zero failures.

What they do

Rough-terrain legged controllers have usually been state machines that trigger reflexes from estimated contact and slip events. This paper trains a blind controller for ANYmal quadrupeds with RL in simulation, using only joint encoders and an IMU. Training uses only rigid, procedurally generated hills, steps and stairs. The same controller was deployed zero-shot on two robot generations on mud, snow, moss, rubble, thick vegetation and running water, with no tuning.

How it works

Teacher. An MLP policy trained with TRPO sees privileged simulator state: height scans around each foot, contact states and forces, friction and external forces. It outputs leg frequencies and foot-position residuals that modulate periodic foot trajectory generators, tracked with analytic IK and joint PD control.
Student. A temporal convolutional network reads up to 100 steps (2 s) of proprioceptive history. It is trained with DAgger to match both the teacher’s action and its latent terrain embedding.
Adaptive curriculum. A particle filter keeps terrain parameters where the policy’s traversability is between 0.5 and 0.9, so terrains stay hard but passable.
Sim-to-real. A learned model of the joint actuators, plus randomized friction, disturbances and observation noise.

Results

Against a model-based baseline in the field: 0.452 vs 0.199 m/s on moss and 0.338 vs 0.197 m/s on mud, with lower cost of transport. The baseline failed often in the forest; the learned controller never did.
DARPA SubT Urban Circuit: two ANYmal-B robots, four 60-minute missions, zero failures.
With an unmodeled 10 kg payload (22.7% of robot weight) it still climbs steps up to 13.4 cm; the baseline climbs none. Its heading error stays within 10°, while the baseline reaches about 30° sideways.
Ablations: direct RL without the teacher fails to walk on slopes or steps. Compared with 20 ms of memory, 2 s handles higher steps and cuts deviation under a 50 N push by 35.5%. Uniform terrain sampling instead of the curriculum lowers success.

Limitations

Trot gait only. Being blind, it must feel out terrain with its body, so the gait is conservative, and it would walk off a cliff if commanded to. The authors suggest adding exteroception.

Videos

Figures

From Figure 1: The learned controller deployed in a variety of challenging natural environments.

From Figure 4: Two-stage training, where an RL teacher with privileged information is imitated by a proprioceptive st... — From Figure 4: Two-stage training, where an RL teacher with privileged information is imitated by a proprioceptive student; an adaptive terrain curriculum that uses particle filtering to keep terrains challenging but traversable; and the controller, where the policy modulates motion primitives via kinematic residuals.

From Figure 3: Indoor tests: walking over loose boards that dislodge underfoot, a foot-trapping reflex over a 16.8 cm... — From Figure 3: Indoor tests: walking over loose boards that dislodge underfoot, a foot-trapping reflex over a 16.8 cm step, and step success rates, speed and heading error against a model-based baseline, with and without a 10 kg payload.

Abstract

Some of the most challenging environments on our planet are accessible to quadrupedal animals but remain out of reach for autonomous machines. Legged locomotion can dramatically expand the operational domains of robotics. However, conventional controllers for legged locomotion are based on elaborate state machines that explicitly trigger the execution of motion primitives and reflexes. These designs have escalated in complexity while falling short of the generality and robustness of animal locomotion. Here we present a radically robust controller for legged locomotion in challenging natural environments. We present a novel solution to incorporating proprioceptive feedback in locomotion control and demonstrate remarkable zero-shot generalization from simulation to natural environments. The controller is trained by reinforcement learning in simulation. It is based on a neural network that acts on a stream of proprioceptive signals. The trained controller has taken two generations of quadrupedal ANYmal robots to a variety of natural environments that are beyond the reach of prior published work in legged locomotion. The controller retains its robustness under conditions that have never been encountered during training: deformable terrain such as mud and snow, dynamic footholds such as rubble, and overground impediments such as thick vegetation and gushing water. The presented work opens new frontiers for robotics and indicates that radical robustness in natural environments can be achieved by training in much simpler domains.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong