End-to-End Training of Deep Visuomotor Policies

Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel

★ Must read April 2015

TL;DR Train a 92,000-parameter CNN that maps raw camera images straight to PR2 motor torques, using guided policy search to turn policy learning into supervised learning. End-to-end training beats policies built on separately trained vision on all four real tasks, e.g. 88.9% vs 55.6% success at screwing on a bottle cap.

What they do

Robot policy search usually sits on top of a hand-built vision pipeline that is trained separately and never adapted to the task. This paper asks whether training perception and control jointly works better. Its policies map monocular RGB images and joint state directly to 7-DoF torques on a PR2 at 20 Hz, for hanging a coat hanger, inserting a block into a shape-sorting cube, fitting a toy hammer’s claw under a nail, and screwing a cap onto a bottle.

How it works

Guided policy search. A trajectory-centric RL method fits time-varying linear-Gaussian controllers for a few target positions, using full state (the robot’s other arm holds the target object) and locally fitted dynamics. The CNN is trained by supervised learning to match them, and a BADMM formulation alternates the two until they agree. Only the local controllers run on the robot during training.
Spatial softmax. Three convolutional layers without pooling, then a softmax over each of 32 response maps and an expected-position step that yields 2D feature points. These join the robot configuration in fully connected layers that output torques; about 92,000 parameters.
Pretraining. The vision layers first regress the target’s 3D pose from about 1000 images, and the controllers get 15 iterations alone before end-to-end training.

Results

End-to-end success (training / new positions / visual distractors): coat hanger 100/100/100%, shape cube 96.3/91.7/87.5%, hammer 91.1/86.7/78.3%, bottle cap 88.9/83.3/62.5%. Frozen pose-estimation features reach 27.5-88.9%; feeding the predicted pose to the controller scores 0% on the cube and 8.9-18.3% on the hammer.
Spatial softmax localizes the target with 1.30 cm error, versus 2.59-4.75 cm for standard CNN heads.
Each policy needs 156-288 trials and 3-4 hours of training, about 15 minutes of it on the robot.

Limitations

Full state must be known during training, which here means placing targets at known positions; freely moving objects would need motion capture. Policies tolerate distractors set apart from the target but fail under drastic backdrop changes or when clutter touches or occludes the object. The feature points assume each feature appears exactly once per image.

Videos

Figures

From Figure 1: The policies use camera images to set motor torques directly on a PR2 robot, shown here at the shape-s... — From Figure 1: The policies use camera images to set motor torques directly on a PR2 robot, shown here at the shape-sorting cube task.

From Figure 2: Visuomotor policy architecture. Three convolutional layers feed a spatial softmax and an expected-posi... — From Figure 2: Visuomotor policy architecture. Three convolutional layers feed a spatial softmax and an expected-position layer that turn pixel features into feature points; these are joined with the robot configuration and passed through three fully connected layers to produce torques.

From Figure 10: Feature points tracked by the policy in the bottle cap task. They sit on the target bottle and the ro... — From Figure 10: Feature points tracked by the policy in the bottle cap task. They sit on the target bottle and the robot gripper and arm, and ignore a distractor bottle in the background that was absent during training.

Abstract

Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a partially observed guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong