Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Kurtland Chua, Roberto Calandra, Rowan McAllister, Sergey Levine

Seminal May 2018

TL;DR PETS models the dynamics with an ensemble of probabilistic neural networks and plans through it with sampled particle trajectories and MPC. It reaches model-free asymptotic performance on four MuJoCo tasks in under 100 trials, with 8x fewer samples than SAC and 125x fewer than PPO on half-cheetah.

What they do

Model-based RL with neural-network dynamics models is sample-efficient but usually converges to worse policies than model-free RL, partly because the networks overfit the little data available early on. PETS (probabilistic ensembles with trajectory sampling) keeps model uncertainty in both learning and planning. It learns only a dynamics model, no policy, and controls with MPC.

How it works

  • Model. An ensemble of 5 bootstrapped networks, each outputting a Gaussian over the next state change. Each network’s variance captures aleatoric noise; disagreement between networks captures epistemic uncertainty from lack of data. Output variance is clipped to the range seen in training.
  • Propagation. 20 particles are rolled out through the ensemble. In TS∞ each particle keeps one bootstrap for the whole horizon; in TS1 it resamples one every step.
  • Planning. CEM searches for the action sequence with the highest expected reward over the particles; the first action is applied and the search repeats at the next step.
  • Tasks. Four MuJoCo tasks: cartpole swing-up, reaching and pushing with a simulated 7-DOF PR2 arm, and half-cheetah.

Results

  • PETS reaches PPO’s asymptotic performance in under 100 trials on all four tasks; on half-cheetah it needs 8x fewer samples than SAC and 125x fewer than PPO.
  • It substantially exceeds the asymptotic performance of the deterministic model-based method of Nagabandi et al. A GP baseline with moment matching does slightly better on cartpole but was infeasible to run on the higher-dimensional tasks.
  • Ablations: probabilistic ensembles are best on every task except cartpole, ahead of single probabilistic networks and deterministic ensembles; a single deterministic network is worst. The propagation method matters less, though moment matching is unreliable on the higher-dimensional half-cheetah.

Limitations

Tested only on four simulated tasks. Running CEM-MPC at every step is expensive at test time, and the authors’ attempts to learn a policy by backpropagating through the uncertainty-aware models did not work. Epistemic uncertainty is separated out but not yet used for exploration.

Videos

Full video (YouTube)

Figures

From Figure 1: PETS. A probabilistic ensemble models the dynamics (bootstrap disagreement captures epistemic uncertai...
From Figure 1: PETS. A probabilistic ensemble models the dynamics (bootstrap disagreement captures epistemic uncertainty, each network's output variance captures aleatoric uncertainty). Trajectory sampling propagates particles through it up to horizon T, and MPC applies the first action of the best sequence and replans.
From Figure A.5: Two probabilistic bootstraps fit to a noisy sine. Each predicts the observation noise; they agree ne...
From Figure A.5: Two probabilistic bootstraps fit to a noisy sine. Each predicts the observation noise; they agree near the data and disagree far from it, which is the model's epistemic uncertainty.
From Figure 3: Learning curves on cartpole, 7-DOF pusher, 7-DOF reacher and half-cheetah. PETS learns every task in u...
From Figure 3: Learning curves on cartpole, 7-DOF pusher, 7-DOF reacher and half-cheetah. PETS learns every task in under 100 trials and outperforms all baselines except the GP model on cartpole.
Abstract

Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance. This is especially true with high-capacity parametric function approximators, such as deep networks. In this paper, we study how to bridge this gap, by employing uncertainty-aware dynamics models. We propose a new algorithm called probabilistic ensembles with trajectory sampling (PETS) that combines uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation. Our comparison to state-of-the-art model-based and model-free deep RL algorithms shows that our approach matches the asymptotic performance of model-free algorithms on several challenging benchmark tasks, while requiring significantly fewer samples (e.g., 8 and 125 times fewer samples than Soft Actor Critic and Proximal Policy Optimization respectively on the half-cheetah task).

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.