Learning agile and dynamic motor skills for legged robots

Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, Marco Hutter

★ Must read January 2019

TL;DR Train ANYmal's controllers with RL in a simulator that uses a neural network, fitted to real robot data, to model the robot's series-elastic actuators, then deploy the policies directly on the robot. The learned locomotion policy tracks commands more accurately and with less torque than the best model-based controller, reaches 1.5 m/s (25% above the previous record), and gets the robot up after a fall.

What they do

ANYmal is a 32 kg quadruped with 12 series-elastic actuators, which no analytical model captures accurately enough for sim-to-real. The authors learn the actuator dynamics from robot data and train RL policies in a hybrid simulator. Three skills (command following, high-speed running, fall recovery) go straight to hardware.

How it works

Rigid-body simulation. A hard-contact solver, with 30 ANYmal models whose link masses, centers of mass and joint positions are randomized (masses by up to ±15%, the others by up to ±2 cm).
Actuator net. A 3×32 softsign MLP maps the current and two past (t−0.01 s, t−0.02 s) joint position errors and velocities to torque. It is trained on under 4 minutes of robot data collected with sine-wave foot trajectories. Validation error is 0.740 Nm, against 3.55 Nm for an ideal actuator model.
Policy. A 256-128 tanh MLP maps base and joint state, joint history, previous action and command to joint position targets for a PD loop. It is trained with default-parameter TRPO, with a curriculum that ramps up the cost terms and noise added to velocity observations.
Compute. The simulator runs about 1000x faster than real time. Locomotion trains in 4 h and recovery in 11 h on one CPU and one GPU. Inference takes 25 µs on one CPU thread.

Results

Random commands: linear velocity error is 0.143 m/s against 0.231 m/s for the model-based controller, and yaw rate error is 0.174 against 0.278 rad/s. Mean torque is 8.23 against 11.7 Nm and power 78.1 against 97.3 W.
Forward running: velocity error is 1.5-2.5x lower than the prior dynamic lateral walk and 5-7x lower than its flying trot, with 23-36% less torque.
High speed: 1.5 m/s on the robot (previous record 1.2 m/s), using the full 40 Nm torque and 12 rad/s joint speed.
Recovery succeeded from all nine tested poses, including one nearly upside down. Policies trained with ideal or analytical actuator models could not take a single step without falling.

Limitations

The cost function and initial-state distribution have to be tuned for each task: about two days for locomotion and a week for recovery. Each network learns only one skill. Actuators are modeled independently, which may not hold for coupled hydraulics. The height estimator assumes flat terrain.

Videos

Figures

From Figure 1: The four steps. Identify the robot's physical parameters and their uncertainty, train an actuator net ... — From Figure 1: The four steps. Identify the robot's physical parameters and their uncertainty, train an actuator net on real data, train a policy with RL in simulation, then deploy it directly on the robot.

From Figure 5: The training loop. The policy net maps the observation and joint state history to joint position targe... — From Figure 5: The training loop. The policy net maps the observation and joint state history to joint position targets; the actuator net maps the joint state history (velocities and position errors) to joint torques for the rigid-body simulator.

From Figure 2: The learned locomotion controller. The gait it discovered for a 1.0 m/s command, its base velocity tra... — From Figure 2: The learned locomotion controller. The gait it discovered for a 1.0 m/s command, its base velocity tracking, and its velocity error, power and torque compared with the best existing controller at commands of 0.25 to 1.0 m/s.

From Figure 4: The learned recovery controller on the real robot gets up from a random initial configuration in under... — From Figure 4: The learned recovery controller on the real robot gets up from a random initial configuration in under 3 seconds.

Abstract

Legged robots pose one of the greatest challenges in robotics. Dynamic and agile maneuvers of animals cannot be imitated by existing methods that are crafted by humans. A compelling alternative is reinforcement learning, which requires minimal craftsmanship and promotes the natural evolution of a control policy. However, so far, reinforcement learning research for legged robots is mainly limited to simulation, and only few and comparably simple examples have been deployed on real systems. The primary reason is that training with real robots, particularly with dynamically balancing systems, is complicated and expensive. In the present work, we introduce a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system, thereby leveraging fast, automated, and cost-effective data generation schemes. The approach is applied to the ANYmal robot, a sophisticated medium-dog-sized quadrupedal system. Using policies trained in simulation, the quadrupedal machine achieves locomotion skills that go beyond what had been achieved with prior methods: ANYmal is capable of precisely and energy-efficiently following high-level body velocity commands, running faster than before, and recovering from falling even in complex configurations.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong