Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, Wojciech Zaremba

Seminal February 2018

TL;DR OpenAI adds 13 goal-conditioned, sparse-reward MuJoCo tasks to Gym for a Fetch arm and a Shadow Dexterous Hand, and benchmarks DDPG with and without Hindsight Experience Replay. DDPG+HER with sparse rewards does best, while plain DDPG often learns nothing on the hand tasks.

What they do

This technical report has two parts. The first is a suite of simulated robotics tasks released in OpenAI Gym. A 7-DoF Fetch arm pushes, slides and picks and places objects, and a 24-DoF Shadow Dexterous Hand reaches with its fingertips and reorients a block, an egg or a pen. Every task is goal-conditioned and has a sparse binary reward. The second part is a list of open research problems, most of them about Hindsight Experience Replay (HER).

How it works

  • Tasks. The tasks run in MuJoCo at 25 Hz. Fetch actions are 4-D (gripper motion plus open/close), and a goal is a 3-D position reached within 5 cm. Hand actions are 20-D joint positions, and an object goal is a position plus a quaternion, met within 1 cm (5 cm for the pen) and 0.1 rad. Variants range from rotation about one axis to full position plus rotation.
  • Goal API. A new gym.GoalEnv returns observation, desired_goal and achieved_goal. It also exposes the reward function, so the reward can be recomputed for a substituted goal, which HER needs.
  • Baselines. DDPG with and without HER, each with sparse and dense rewards. Training uses 19 CPU cores and 5 seeds. One hyperparameter set, picked from 40 random samples on HandManipulateBlockRotateZ, is used for every task.

Results

  • All four configurations solve FetchReach. On the other Fetch tasks, DDPG+HER clearly beats the rest within 50 epochs (4.75 million timesteps).
  • On the hand tasks (block, egg and pen get 200 epochs, 38 million timesteps), plain DDPG often fails to learn at all. HER learns partly successful policies on every task, but HandManipulatePen is not fully solved.
  • With HER, sparse rewards work better than dense ones. The authors’ explanation: a sparse return is easier for the critic to learn, and a dense reward pushes the policy toward one particular strategy.

Limitations

Simulation only, with DDPG variants as the only baselines. The authors note that HER changes the distribution of replayed transitions in an unprincipled way, which could in theory break training in very stochastic environments, though they have not seen this in practice.

Videos

Full video (YouTube)

Figures

From Figure 1: FetchPickAndPlace, one of the four Fetch arm environments. The robot must grasp a box and move it to a...
From Figure 1: FetchPickAndPlace, one of the four Fetch arm environments. The robot must grasp a box and move it to a target on the table or in the air above it.
From Figure 2: HandManipulateBlock, one of the four Shadow Dexterous Hand environments. The hand must bring a block l...
From Figure 2: HandManipulateBlock, one of the four Shadow Dexterous Hand environments. The hand must bring a block lying on its palm to a target pose.
From Figure 3: Median test success rate on FetchPickAndPlace over 50 epochs for DDPG and DDPG+HER with sparse and den...
From Figure 3: Median test success rate on FetchPickAndPlace over 50 epochs for DDPG and DDPG+HER with sparse and dense rewards. Shaded areas show the interquartile range.
Abstract

The purpose of this technical report is two-fold. First of all, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.