Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Learning to Walk via Deep Reinforcement Learning

Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, Sergey Levine

Seminal December 2018

TL;DR Soft actor-critic with its temperature tuned automatically toward a target entropy, used to teach a Minitaur quadruped to walk from scratch on the real robot. A gait emerges in about two hours (about 400 rollouts) and handles a slope, wooden blocks and steps it never trained on.

What they do

Deep RL on real legged robots is held back by sample cost and by hyperparameters that cannot be tuned through trial and error without breaking hardware. The authors extend soft actor-critic (SAC) so that its most sensitive setting, the entropy temperature, is learned during training, and use it to train a Minitaur quadruped to walk directly on the robot, with no simulator and no dynamics model.

How it works

  • Entropy-constrained objective. Maximize return subject to a minimum expected entropy, so entropy can still vary between states. The temperature is the Lagrange multiplier of this constraint and is updated by gradient steps alongside the Q-functions and policy. On the benchmarks the target is simply -1 per action dimension.
  • Asynchronous system. The robot’s on-board computer runs rollouts, motion capture supplies the forward-progress reward, and a workstation trains; each job can be restarted alone after hardware errors.
  • Robot setup. Observations stack the last five steps of motor angles, base roll and pitch, their rates, and actions (112 dimensions). Actions are leg swing and extension, tracked by a low-gain PD controller. The reward also penalizes joint acceleration, base roll and front legs folding under the body.

Results

  • The real robot learns to walk from 160k control steps (about 400 rollouts of up to 10 s), about two hours. It walks at 0.32 m/s with a periodic, synchronized gait.
  • Never trained with pushes or obstacles, it recovered from pushes in 10/10 tests and crossed a slope, wooden blocks and steps in 10/10 trials each.
  • In simulation the Minitaur policy withstands 220 N lateral pushes, against 130 N for the best of 1000 PPO policies.
  • On Gym benchmarks it matches or beats SAC with a per-environment tuned reward scale, as well as DDPG, TD3 and PPO. Standard SAC needs a reward scale roughly two orders of magnitude larger on simulated Minitaur than on the benchmarks.

Limitations

Training needs manual resets between episodes and motion capture for the reward, and there is no safety layer, which the authors say is needed before learning on larger robots. Real-robot returns are slightly below simulation, and the truncated dual gradient updates are not guaranteed to reach the optimum.

Videos

Full video (YouTube)
Full video (YouTube)

Figures

From Figure 1: The gait was learned on flat ground only; here the robot walks on a slope it never saw during training.
From Figure 1: The gait was learned on flat ground only; here the robot walks on a slope it never saw during training.
From Figure 2: The learning system. Data collection on the robot, reward computation from motion capture and training...
From Figure 2: The learning system. Data collection on the robot, reward computation from motion capture and training on a workstation run asynchronously on separate machines, linked by a replay buffer.
From Figure 4: Normalized return of the method as a function of target entropy on HalfCheetah, Ant, Walker2d and simu...
From Figure 4: Normalized return of the method as a function of target entropy on HalfCheetah, Ant, Walker2d and simulated Minitaur. The range of good values is about the same for all four tasks.
From Figure 7: Learning curve of SAC with learned temperature on the real Minitaur robot.
From Figure 7: Learning curve of SAC with learned temperature on the real Minitaur robot.
Abstract

Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions. In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics. Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters. While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning. In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies. We apply this method to learning walking gaits on a real-world Minitaur robot. Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment. We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Videos of training and the learned policy can be found on the project website.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.