Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
TL;DR Soft actor-critic (SAC) is an off-policy actor-critic that trains a stochastic policy to maximize reward plus entropy. On six continuous-control benchmarks it matches DDPG, PPO, SQL and TD3 on the easier tasks and beats them on the hardest, including the 21-action-dimension rllab Humanoid, where DDPG makes no progress.
What they do
Off-policy deep RL methods such as DDPG reuse past data but are brittle and sensitive to hyperparameters. On-policy methods such as PPO are more stable but need new samples for every gradient step. Soft actor-critic (SAC) is an off-policy actor-critic that maximizes expected reward plus the entropy of a stochastic policy, aiming for both sample efficiency and stability.
How it works
- Soft policy iteration. In the tabular case, alternate soft policy evaluation (a Bellman backup that includes the policy’s entropy) with policy improvement (a KL projection of the policy onto the exponential of the Q-function). This provably converges to the best maximum-entropy policy in the chosen policy class.
- Practical algorithm. A soft value network, two soft Q-networks (their minimum is used) and a Gaussian policy are trained with SGD on replay-buffer samples, with an exponentially averaged target value network (τ = 0.005 on every task). The policy minimizes E[log π − Q] through the reparameterization trick, and a tanh squashes actions into bounds.
- Temperature. The reward scale sets the trade-off between reward and entropy. It is the only hyperparameter tuned per task.
Results
- Six benchmarks (five OpenAI gym tasks and the 21-action-dimension rllab Humanoid), five seeds each: SAC is comparable to the baselines on the easier tasks and ahead in learning speed and final return on the harder ones. DDPG makes no progress on Ant-v1, Humanoid-v1 or Humanoid (rllab). SQL learns every task but more slowly than SAC, and finishes lower.
- Over five seeds on Humanoid (rllab), a deterministic variant without entropy maximization varies far more between runs than SAC.
- The reward scale is 5 on four tasks, 20 on Humanoid-v1 and 10 on Humanoid (rllab). Too small a scale gives a near-uniform policy. Too large a scale gives a near-deterministic one that gets stuck in poor local minima.
Limitations
The paper has no limitations section. All experiments are simulated benchmarks, and results are given only as learning curves. The convergence proof covers only finite action spaces, and the deep-network algorithm approximates it. SAC is particularly sensitive to reward scale, which must be tuned per environment.
Videos
Figures
Abstract
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.