Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Continuous control with deep reinforcement learning

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra

Seminal September 2015

TL;DR DDPG adds DQN's replay buffer and target networks, plus batch normalization, to a deterministic actor-critic so it can learn continuous control. With one set of hyperparameters it solves more than 20 simulated physics tasks, often from raw pixels, nearly all within 2.5 million steps, and sometimes beats an iLQG planner that has the true dynamics.

What they do

DQN needs a max over discrete actions; discretizing a 7-DoF arm with three levels per joint already gives 2187 actions. DDPG builds instead on the deterministic policy gradient (DPG): an actor network maps states to continuous actions and a critic learns Q(s, a). The paper adds the tricks that stabilized DQN and runs one algorithm and set of hyperparameters on more than 20 simulated MuJoCo tasks and the Torcs racing game, from low-dimensional states and from raw pixels.

How it works

  • Actor-critic. The critic is trained on Bellman targets as in Q-learning. The actor is updated with the deterministic policy gradient, the chain rule applied to the critic’s estimate of return.
  • Replay and soft targets. Transitions go into a 10^6-sample replay buffer. Slowly tracking copies of both actor and critic (tau = 0.001) compute the targets, which keeps critic training from diverging.
  • Batch normalization on the state input and hidden layers, so the same hyperparameters work across observations with different units and scales.
  • Exploration adds temporally correlated Ornstein-Uhlenbeck noise to the actor’s output. From pixels, each action is repeated for 3 simulator steps and the three 64x64 frames are stacked to expose velocities.

Results

  • Scores are normalized so a random policy gets 0 and an iLQG planner with full access to the dynamics gets 1, averaged over 5 replicas. From states DDPG averages 0.946 on pendulum, 0.903 on cheetah and 0.655 on gripper. On hardCheetah it averages 1.311 from states and 1.204 from pixels, beating the planner.
  • Pixel policies are close to state-based ones on some tasks (fixedReacherDouble 0.872 vs 0.924) and far behind on others (hopper 0.112 vs 0.676).
  • Without target networks learning is poor: DPG with replay and batch normalization averages -0.008 on cheetah.
  • Nearly all tasks are solved within 2.5 million steps, 20 times fewer than DQN needs on Atari.

Limitations

Like most model-free RL, DDPG needs many training episodes, and nonlinear function approximators remove any convergence guarantee. Q estimates are accurate on simple tasks but worse on harder ones. On Torcs some replicas learned to complete a lap while others failed. All experiments are in simulation.

Figures

From Figure 1: Some of the environments, from left: cartpole swing-up, a reaching task, a grasp-and-move task, a puck...
From Figure 1: Some of the environments, from left: cartpole swing-up, a reaching task, a grasp-and-move task, a puck-hitting task, a monoped balancing task, two locomotion tasks and the Torcs driving simulator.
From Figure 2: Normalized reward over training (million steps) on ten tasks for variants of DPG: original DPG with ba...
From Figure 2: Normalized reward over training (million steps) on ten tasks for variants of DPG: original DPG with batch normalization (light grey), with a target network (dark grey), with target networks and batch normalization (green), and with target networks from pixels only (blue). Target networks are crucial.
From Figure 3: Estimated Q values versus observed returns on test episodes from 5 replicas; the dotted line is unity....
From Figure 3: Estimated Q values versus observed returns on test episodes from 5 replicas; the dotted line is unity. Estimates are accurate on pendulum and cartpole and less accurate on cheetah.
Abstract

We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.