Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection

Sergey Levine, Peter Pastor, Alex Krizhevsky, Deirdre Quillen

★ Must read March 2016

TL;DR Train a CNN on about 800,000 self-supervised grasp attempts from up to 14 robots to predict whether a gripper motion will lead to a successful grasp, then use it to servo the gripper from an uncalibrated monocular camera. When clearing a bin of novel objects it fails on 17.5% of the first 30 grasps, against 33.7% for an open-loop variant and 50.8% for a hand-engineered depth-based system.

What they do

Grasping usually relies on careful camera-to-robot calibration and open-loop planning. This paper learns hand-eye coordination instead: a CNN predicts how likely a task-space gripper motion is to end in a successful grasp, using only monocular images. A servoing loop re-queries the network continuously, so the robot corrects its own errors without knowing where the camera is.

How it works

  • Data. Over two months, 6 to 14 arms with over-the-shoulder cameras made about 800,000 grasp attempts with no human labels. Camera poses varied and grippers wore unevenly. Success labels came from the finger opening plus a drop test that compares bin images. About half the attempts were random, the rest used the latest network.
  • Predictor. The network takes the current image, a pre-grasp image without the gripper, and a motion vector (3D translation plus rotation about the vertical axis) in the robot frame. Each training sample pairs an image with the vector from the current pose to the final pre-grasp pose and the grasp outcome.
  • Servoing. Three iterations of the cross-entropy method (64 samples, top 6) pick the best motion at about 2 to 5 Hz. The gripper closes when not moving scores at least 90% of the best motion, and lifts when it scores below 50%.

Results

  • Without replacement on unseen objects, failure rates over the first 10/20/30 grasps are 10.0/17.5/17.5%, against 27.5/38.7/33.7% open loop, 32.5/35.0/50.8% hand-designed and 67.5/70.0/72.5% random.
  • With replacement (100 attempts) it fails 20%, against 43% open loop and 35% hand-designed.
  • Less data hurts: with the first 12% of the dataset, failure over the first 30 grasps is 47.5%, and 25.0% with half.

Limitations

It would not readily transfer to robots that differ substantially from the training ones, and since all training grasps were on flat surfaces it is unlikely to work on shelves or in cubbies. The action representation assumes intermediate motions don’t move objects, the close and lift rules are ad hoc, and the workspace and table height are set by hand.

Videos

Full video (YouTube)

Figures

From Figure 1: The data collection setup of 14 robotic manipulators, used to gather over 800,000 grasp attempts.
From Figure 1: The data collection setup of 14 robotic manipulators, used to gather over 800,000 grasp attempts.
From Figure 2: A network input image overlaid with sampled target grasp positions, colored by predicted success from ...
From Figure 2: A network input image overlaid with sampled target grasp positions, colored by predicted success from green (1.0) to red (0.0). The network itself only receives offsets from the current gripper position in the robot frame.
From Figure 4: The CNN grasp predictor. The current image and a pre-grasp image pass through convolution and pooling ...
From Figure 4: The CNN grasp predictor. The current image and a pre-grasp image pass through convolution and pooling layers; the motor command goes through one fully connected layer and is tiled and added to the response map; further convolutions and two fully connected layers output the probability of a successful grasp.
From Figure 10: Difficult objects grasped by the method, including translucent, awkwardly shaped and heavy ones.
From Figure 10: Difficult objects grasped by the method, including translucent, awkwardly shaped and heavy ones.
Abstract

We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. To train our network, we collected over 800,000 grasp attempts over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and hardware. Our experimental evaluation demonstrates that our method achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.