Addressing Function Approximation Error in Actor-Critic Methods

Scott Fujimoto, Herke van Hoof, David Meger

Seminal February 2018

TL;DR TD3 curbs value overestimation in DDPG with three changes, a target that takes the minimum of two critics, less frequent actor updates and noise on the target action. Across seven MuJoCo tasks it matches or beats DDPG, PPO, TRPO, ACKTR and SAC, and on Ant it reaches a return of 4372 where no baseline exceeds 1822.

What they do

Q-learning overestimates values when its targets are noisy. The paper proves that, under basic assumptions, deterministic policy gradient updates overestimate too, and measures a clear bias in DDPG’s value estimates on Hopper and Walker2d. Double DQN’s fix does little here, because the slowly changing policy keeps current and target networks too similar. TD3 (Twin Delayed DDPG) adds three changes to a re-tuned DDPG.

How it works

Clipped Double Q-learning. Two critics regress to one shared target built from the minimum of their target estimates. Any added bias is toward underestimation, which the policy update does not propagate.
Delayed policy updates. The actor and target networks update only every d = 2 critic updates (tau = 0.005), so the policy follows a lower-variance value estimate. On Hopper, values diverge without slow target networks once the policy learns.
Target policy smoothing. Gaussian noise (sigma 0.2, clipped to ±0.5) on the target action makes similar actions take similar values.

Results

Max average return over 10 seeds and 1M steps: HalfCheetah 9637 (DDPG 3306, best other 8577), Hopper 3564 (SAC 2997), Walker2d 4683 (PPO 3318), Ant 4372 (ACKTR 1822). On the two inverted-pendulum tasks results are close to DDPG’s.
Ablations: removing Clipped Double Q drops Ant from 4185 to 850 and Walker2d from 4565 to 2579. Actor-critic Double Q-learning reaches 2446 on Ant, Double DQN 1092.
Re-tuning alone already lifts DDPG on HalfCheetah (3163 to 8401 final return).

Limitations

The minimum can underestimate values. Each component’s benefit varies by task, and adding just one to the re-tuned DDPG is mostly insignificant. The convergence proof covers only tabular, finite MDPs, and all experiments are simulated. SAC was an older version run at one update per step instead of four; the authors note its newer version, which adopts Clipped Double Q, is competitive on most tasks.

Figures

From Figure 4: The HalfCheetah-v1 MuJoCo environment, one of the OpenAI Gym tasks used for evaluation.

From Figure 4: The Ant-v1 MuJoCo environment.

Abstract

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong