Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Hindsight Experience Replay

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba

Seminal July 2017

TL;DR Replay each failed episode as if its goal had been the state the agent actually reached, so sparse binary rewards still give a learning signal. On a simulated Fetch arm, DDPG with HER solves pushing, sliding and pick-and-place where plain DDPG solves none, and the pick-and-place policy succeeds 5/5 on the real robot.

What they do

Robot RL usually needs a carefully shaped reward. Hindsight Experience Replay (HER) learns from a binary success signal instead. A failed attempt to reach goal g still shows how to reach the state the agent actually ended up in, so HER also stores the episode with that state as the goal. It works with any off-policy algorithm.

How it works

  • Goal-conditioned learning. Policy and Q-function take a goal as input (as in UVFA). Here the goal is a 3D object position, and the reward is -1 at every step until the object is within a fixed tolerance of it.
  • Hindsight replay. Each transition goes into the replay buffer with the original goal and again with substitute goals, with the reward recomputed. The goal does not change the dynamics, so this relabeled data is valid off-policy. Strategies: the episode’s final state, or k states reached later in the same episode (future).
  • Setup. DDPG controls a simulated 7-DoF Fetch arm in MuJoCo with a 4-D action (gripper displacement plus finger opening) on pushing, sliding and pick-and-place.
  • Real robot. A CNN trained on domain-randomized renders estimates the box position from the head camera. The policy runs in a simulator initialized from that estimate, and its joint angles are sent to the real arm.

Results

  • Bit flipping: DQN alone solves only up to 13 bits; with HER it solves up to 50.
  • Plain DDPG solves none of the three tasks and count-based exploration helps only on sliding. DDPG+HER solves all three almost perfectly.
  • Shaped distance rewards fail on every task, with or without HER. Future replay with k = 4 or 8 works best and is the only strategy that nearly solves sliding; k above 8 hurts.
  • On the physical robot, pick-and-place first succeeds 2/5. After retraining with 1 cm observation noise it succeeds 5/5.

Limitations

HER needs an off-policy learner and a known map from states to achieved goals. Pick-and-place only trained once half the episodes started from a recorded grasped state, and pushing needed an extra reward term against MuJoCo object penetration. Gripper rotation is fixed, and the real-robot test is only 5 trials.

Figures

From Figure 2: the three tasks, pushing (top row), sliding (middle row) and pick-and-place (bottom row). The red ball...
From Figure 2: the three tasks, pushing (top row), sliding (middle row) and pick-and-place (bottom row). The red ball is the goal position.
From Figure 3: success rate over training on the multi-goal tasks for DDPG, DDPG with count-based exploration, and DD...
From Figure 3: success rate over training on the multi-goal tasks for DDPG, DDPG with count-based exploration, and DDPG+HER with the final (blue) and future, k = 4 (red) strategies, averaged over 5 seeds.
From Figure 6: ablation of how replay goals are chosen (final, random, episode, future) and how many per transition; ...
From Figure 6: ablation of how replay goals are chosen (final, random, episode, future) and how many per transition; top row is the highest test success rate during training, bottom row the average.
From Figure 7: the pick-and-place policy deployed on the physical Fetch robot.
From Figure 7: the pick-and-place policy deployed on the physical Fetch robot.
Abstract

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.