Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates

Shixiang Gu, Ethan Holly, Timothy Lillicrap, Sergey Levine

Seminal October 2016

TL;DR Asynchronous NAF trains one deep Q-function from a replay buffer that several robots fill in parallel. Two 7-DoF arms learn to pull open a door from scratch, without demonstrations, in about 2.5 hours, and the final policy succeeds in 20 of 20 trials.

What they do

Deep RL had mostly been limited to simulation because of its sample cost. This paper shows that off-policy deep Q-learning (NAF, plus DDPG in simulation) is efficient enough to learn manipulation on real 7-DoF arms from scratch, with no demonstrations and no task-specific policy representation, and that pooling experience from several robots speeds it up further.

How it works

  • Asynchronous NAF. One trainer thread on a central server updates the Q-function from a shared replay buffer at about 100 updates per second. A collector thread on each robot runs the current policy plus exploration noise at 20 Hz, syncs weights at the start of every episode and pushes its transitions to the buffer.
  • Networks. Two hidden layers of 100 units for each of the policy mean, the advantage matrix (through its Cholesky factor) and the value. Inputs are joint angles, end-effector position, their derivatives and a task target, e.g. the handle position and the IMU quaternion of the door.
  • Safety. Joint velocity and position limits, and a bounding sphere for the end effector: commands that would leave it are projected onto its surface. Contact tasks needed extra heuristics.

Results

  • Simulation (reach, door push, door pull, JACO pick & place): DDPG and NAF reach 100% success on all four. A linear-policy NAF peaks at 5% on door pulling and 40% on door pushing, and needs 8k vs 3.6k episodes (reach) and 12k vs 2.9k (pick & place) to reach 100%.
  • More simulated workers can speed up learning and change final success, up to a limit where extra data stops helping.
  • Real reaching on up to four robots: 2 or 4 workers clearly beat 1, with little gain past 2.
  • Real door pulling: two workers reach 100% success in about 2.5 hours (under 500,000 updates); one worker needs significantly more than 4 hours.

Limitations

Rewards are shaped with distances to the target, handle and open-door pose; with only a binary success signal the tasks become substantially harder. On hardware the door pose was fixed and door pushing could not be tested. Experience from all robots is simply pooled.

Figures

From Figure 1: Two robots learning a door opening task together, training a single shared policy.
From Figure 1: Two robots learning a door opening task together, training a single shared policy.
From Figure 6: A robot hooking the handle and pulling the door open with the policy learned by asynchronous NAF.
From Figure 6: A robot hooking the handle and pulling the door open with the policy learned by asynchronous NAF.
From Figure 5: Success rate on simulated door pushing with 1, 2 and 3 workers; learning speed and final success depen...
From Figure 5: Success rate on simulated door pushing with 1, 2 and 3 workers; learning speed and final success depend on the number of workers.
From Figure 8: Test reward for real-world door opening; two workers learn significantly faster than one.
From Figure 8: Test reward for real-world door opening; two workers learn significantly faster than one.
Abstract

Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinforcement learning algorithms have so far been restricted to simulated settings and relatively simple tasks, due to their apparent high sample complexity. In this paper, we demonstrate that a recent deep reinforcement learning algorithm based on off-policy training of deep Q-functions can scale to complex 3D manipulation tasks and can learn deep neural network policies efficiently enough to train on real physical robots. We demonstrate that the training times can be further reduced by parallelizing the algorithm across multiple robots which pool their policy updates asynchronously. Our experimental evaluation shows that our method can learn a variety of 3D manipulation skills in simulation and a complex door opening skill on real robots without any prior demonstrations or manually designed representations.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.