Learning Agile Robotic Locomotion Skills by Imitating Animals

Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, Sergey Levine

Seminal April 2020

TL;DR A quadruped learns gaits, spins and hops in simulation by imitating retargeted dog mocap and animator clips, then adapts on the real robot by searching a learned latent dynamics space in about 50 trials. The learned dog trot reaches 1.08 m/s, faster than the manufacturer's 0.84 m/s gait.

What they do

Instead of a hand-designed controller or reward per skill, an 18-DoF Laikago quadruped gets a reference motion (dog mocap or an artist’s animation), and RL in simulation trains a policy to track it. A short real-world search over learned latent dynamics then transfers the policy.

How it works

Retargeting. Inverse kinematics maps keypoints on the dog’s feet and hips onto the robot.
Imitation. A 30 Hz policy sees recent joint angles, IMU orientation and actions, plus reference poses up to about 1 s ahead, and outputs PD targets. The reward tracks joint angles and velocities, foot positions and root motion. PPO is trained on about 200 million PyBullet samples.
Latent dynamics. Mass, inertia, motor strength and friction, latency and ground friction are randomized. An encoder turns them into a latent z, and a KL information bottleneck (weight beta) trades robustness against adaptability.
Real-world adaptation. On the robot, advantage-weighted regression searches over z only, using about 50 trials of 5-10 s per policy.

Results

Skills include pacing, trotting, backwards gaits (mocap played in reverse), a spin and a Hop-Turn with a 90 degree midair turn. Dog Trot reaches 1.08 m/s and backwards trot 1.20 m/s, while the manufacturer’s fastest gait reaches 0.84 m/s.
Real-world normalized return after adaptation vs. a robust (randomized, non-adaptive) policy: Dog Pace 0.83 vs 0.35, Dog Spin 0.75 vs 0.21, Dog Backwards Trot 0.66 vs 0.12. The robust policy scores highest on In-Place Steps, Side-Steps and Dog Backwards Pace. Without randomization, most skills fail to transfer.
In out-of-distribution simulated dynamics, adaptive Dog Pace beats 0.6 return in 50% of environments vs 38% for robust. Beta = 1e-4 gave the best trade-off, and bottlenecked encoders generally beat an unconstrained one before and after adaptation.

Limitations

Large jumps and runs were not achieved, and the learned gaits are less stable than the best hand-designed controllers. Running Man was not reproduced: the robot shuffles backwards without lifting its feet.

Videos

Figures

From Figure 2: The three stages. Animal motion data is retargeted to the robot, a policy learns to imitate it in simu... — From Figure 2: The three stages. Animal motion data is retargeted to the robot, a policy learns to imitate it in simulation, and domain adaptation carries it over to the real robot.

From Figure 6: The adaptive policy. An encoder maps the dynamics parameters to a Gaussian over a latent encoding z, w... — From Figure 6: The adaptive policy. An encoder maps the dynamics parameters to a Gaussian over a latent encoding z, which the policy takes alongside the state and goal; a separate network models the value function.

From Figure 5: Average normalized return of each skill on the real robot, for policies trained without randomization,... — From Figure 5: Average normalized return of each skill on the real robot, for policies trained without randomization, robust policies, and adaptive policies before and after adaptation. 15 trials per method.

Abstract

Reproducing the diverse and agile locomotion skills of animals has been a longstanding challenge in robotics. While manually-designed controllers have been able to emulate many complex behaviors, building such controllers involves a time-consuming and difficult development process, often requiring substantial expertise of the nuances of each skill. Reinforcement learning provides an appealing alternative for automating the manual effort involved in the development of controllers. However, designing learning objectives that elicit the desired behaviors from an agent can also require a great deal of skill-specific expertise. In this work, we present an imitation learning system that enables legged robots to learn agile locomotion skills by imitating real-world animals. We show that by leveraging reference motion data, a single learning-based approach is able to automatically synthesize controllers for a diverse repertoire behaviors for legged robots. By incorporating sample efficient domain adaptation techniques into the training process, our system is able to learn adaptive policies in simulation that can then be quickly adapted for real-world deployment. To demonstrate the effectiveness of our system, we train an 18-DoF quadruped robot to perform a variety of agile behaviors ranging from different locomotion gaits to dynamic hops and turns.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong