Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Overcoming Exploration in Reinforcement Learning with Demonstrations

Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel

Seminal September 2017

TL;DR Add 100 VR demonstrations to DDPG with Hindsight Experience Replay through a critic-filtered behavior-cloning loss and resets to demonstration states. A simulated Fetch arm learns to stack 6 blocks (32% success), while behavior cloning and HER alone already get 0% on 4 blocks.

What they do

With a sparse reward, RL rarely finds any reward on long multi-step manipulation, and behavior cloning drifts off the demonstrations. The authors add 100 human demonstrations, teleoperated in VR with an HTC Vive, to DDPG with Hindsight Experience Replay (HER). They test on a simulated 7-DoF Fetch arm in MuJoCo that observes object positions and commands end-effector position and finger opening at 50 Hz: the HER push, slide and pick-and-place tasks, and stacking up to six blocks.

How it works

  • Demonstration buffer. Demonstrations sit in a second replay buffer; every minibatch of 1024 gets 128 extra demo transitions for both actor and critic updates.
  • Behavior cloning loss. An auxiliary squared-error loss on demo transitions pulls the actor toward the demonstrated actions.
  • Q-filter. That loss applies only where the critic scores the demo action above the actor’s own, so the policy can move past imperfect demonstrations.
  • Resets to demonstration states. Some training episodes start from a state sampled from a demonstration, with its final state as the goal. Not used at evaluation.

Results

  • HER tasks: convergence is 4x faster than HER on pushing and 2x on sliding. Without demonstrations or the HER paper’s grasped-block starts, HER never solves pick-and-place; this method does, in about 1 million timesteps.
  • Stacking 3 blocks with a sparse reward: 99% success, against 1% for BC, 0% for HER and 1% for BC-initialized HER, whose untrained critic wipes out the pretrained actor.
  • Resets matter on long tasks: 4 blocks with a sparse reward goes from 1% to 54%, 6 blocks with the step reward from 4% to 32%. They hurt short ones (3 blocks: 89% vs 99%).
  • Without the BC loss, 3-block stacking learns about 2x slower and longer tasks get no success. The Q-filter speeds up 3-block learning but is mixed under the step reward.

Limitations

Simulation only, with object positions given. Hard tasks need an amount of experience that is impractical outside simulation. Demonstrations are required (more than half of the stacking demos were rejected), and resets need a simulator that can be set to arbitrary states. Few seeds: 2 for 4 and 5 blocks, 1 for 6.

Videos

Full video (YouTube)

Figures

From Figure 1: Frames from rollouts of the learned policy. Six blocks start at random positions, and red spheres mark...
From Figure 1: Frames from rollouts of the learned policy. Six blocks start at random positions, and red spheres mark their targets, stacked in order on top of block A.
From Figure 1: More rollout frames of the learned policy building the tower on the red targets.
From Figure 1: More rollout frames of the learned policy building the tower on the red targets.
From Figure 4: Ablations on stacking 3 blocks with a fully sparse reward, 5 seeds per method with the median in bold....
From Figure 4: Ablations on stacking 3 blocks with a fully sparse reward, 5 seeds per method with the median in bold. No HER stays at 0% success; the method without resets learns faster than the ablations, and with resets it starts faster but converges lower.
From Figure 5: Success rate on stacking 6 blocks with the step reward. As the number of blocks grows, resets from dem...
From Figure 5: Success rate on stacking 6 blocks with the step reward. As the number of blocks grows, resets from demonstration states become more important.
Abstract

Exploration in environments with sparse rewards has been a persistent problem in reinforcement learning (RL). Many tasks are natural to specify with a sparse reward, and manually shaping a reward function can result in suboptimal performance. However, finding a non-zero reward is exponentially more difficult with increasing task horizon or action dimensionality. This puts many real-world tasks out of practical reach of RL methods. In this work, we use demonstrations to overcome the exploration problem and successfully learn to perform long-horizon, multi-step robotics tasks with continuous control such as stacking blocks with a robot arm. Our method, which builds on top of Deep Deterministic Policy Gradients and Hindsight Experience Replay, provides an order of magnitude of speedup over RL on simulated robotics tasks. It is simple to implement and makes only the additional assumption that we can collect a small set of demonstrations. Furthermore, our method is able to solve tasks not solvable by either RL or behavior cloning alone, and often ends up outperforming the demonstrator policy.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.