Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov

★ Must read July 2017

TL;DR PPO replaces TRPO's constrained update with a clipped probability-ratio objective that can be optimized for several epochs of minibatch SGD on the same batch. With clipping at 0.2 it scores 0.82 (normalized) on 7 MuJoCo tasks, vs at most 0.74 for KL-penalty variants and -0.39 with no clipping.

What they do

Reusing a batch for several policy gradient steps tends to cause destructively large updates. TRPO limits each update with a KL constraint, but it is relatively complicated and does not fit architectures with noise (dropout) or policy-value parameter sharing. PPO aims for TRPO’s reliability with first-order optimization only: a surrogate objective that removes the incentive to move far from the old policy, so the same data can be reused for several epochs.

How it works

Clipped objective. With probability ratio r = π_new(a

s) / π_old(a

s) and advantage estimate A, each term is min(r·A, clip(r, 1−ε, 1+ε)·A). Changes in r that would improve the objective beyond the clip range are ignored, changes that make it worse are kept, so the result is a pessimistic bound on the unclipped objective.

Adaptive KL penalty, tested as an alternative: penalize KL to the old policy and halve or double the coefficient when the measured KL is 1.5x below or above a target.
Algorithm. N parallel actors each collect T timesteps, advantages come from truncated generalized advantage estimation, and the loss (plus a value-function term when policy and value share a network, optionally an entropy bonus) is optimized for K epochs of minibatch Adam.

Results

Surrogate comparison on 7 MuJoCo tasks (1M timesteps, 3 seeds, normalized score): clipping with ε = 0.2 scores 0.82 (0.76 at 0.1, 0.70 at 0.3); adaptive KL 0.68-0.74; fixed KL 0.62-0.72; no clipping or penalty -0.39.
Against tuned TRPO, CEM, A2C, A2C with trust region and adaptive vanilla PG, PPO does better on almost all of these continuous control tasks.
In Roboschool, PPO learns 3D humanoid running, steering toward moving targets, and getting up while being pelted by cubes.
On 49 Atari games, PPO wins 30 by average reward over all of training (ACER 18, A2C 1) but 19 by reward over the last 100 episodes (ACER 28).

Limitations

The paper has no limitations section. Evidence is empirical only, on simulated MuJoCo and Roboschool tasks and Atari, with hyperparameters searched on a cheap 1M-step benchmark. On final Atari performance ACER wins more games than PPO.

Figures

From Figure 1: One timestep's term of the clipped surrogate objective as a function of the probability ratio r, for p... — From Figure 1: One timestep's term of the clipped surrogate objective as a function of the probability ratio r, for positive advantage (left) and negative advantage (right). The red circle marks the starting point, r = 1.

From Figure 3: Learning curves of PPO (clip), A2C, A2C with trust region, CEM, adaptive vanilla policy gradient and T... — From Figure 3: Learning curves of PPO (clip), A2C, A2C with trust region, CEM, adaptive vanilla policy gradient and TRPO on seven MuJoCo environments over one million timesteps.

From Figure 5: Still frames of a policy learned on RoboschoolHumanoidFlagrun. The humanoid runs toward a target, then... — From Figure 5: Still frames of a policy learned on RoboschoolHumanoidFlagrun. The humanoid runs toward a target, then turns and runs toward a new one when the target moves.

Abstract

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong