Real-Time Grasp Detection Using Convolutional Neural Networks

Joseph Redmon, Anelia Angelova

Seminal December 2014

TL;DR A single AlexNet-style CNN looks once at an RGB-D image and regresses a grasp rectangle directly, instead of scoring many patches with a sliding window. Its MultiGrasp variant reaches 88.0% accuracy on the Cornell Grasping Dataset at 13 frames per second, versus 73.9% at 13.5 s per image for the previous best.

What they do

Grasp detection on the Cornell dataset was done with sliding-window classifiers that score many small patches, taking 13.5 s per image. This paper applies one large CNN once to the whole RGB-D image and predicts a parallel-gripper grasp rectangle (center, width, height, angle) directly, treating grasp detection like object detection with an added orientation term.

How it works

Network. An AlexNet-style model pretrained on ImageNet. Depth replaces the blue channel so the pretrained filters can be reused; without pretraining the models overfit.
Direct regression. Six outputs: position, size, and the sine and cosine of twice the angle (grasps are two-fold symmetric). Each time an image is seen, a random ground-truth grasp is the target, so the model learns an average grasp.
Regression + classification. Extra output neurons predict one of 16 object categories from the same features.
MultiGrasp. The image is split into a 7x7 grid; each cell predicts one grasp plus a heatmap value for how likely it is a good grasp, and the highest-weighted box is the final answer.

Results

Rectangle metric (angle within 30 degrees, Jaccard index above 25%), five-fold cross-validation: direct regression 84.4% image-wise / 84.9% object-wise, MultiGrasp 88.0% / 87.1%, versus 73.9% / 75.6% for Lenz et al.
76 ms per batch of 128 images on a Tesla K20 GPU, reported as 13 fps; over 150 times faster than prior methods.
The combined model classifies objects at 90.0% image-wise and 61.5% object-wise (most-common class: 17.7%) with no loss in detection accuracy.

Limitations

The models assume one graspable object per image. Direct regression averages between good grasps and fails on round objects such as flying discs. Grasps are 2-D rectangles assumed to project back to 3-D; nothing is executed on a robot. MultiGrasp’s multiple-grasp predictions are not evaluated quantitatively, as no dataset supports it.

Figures

From Figure 1: Objects in the Cornell Grasping Dataset, each labelled with several grasps given as oriented 2-D recta... — From Figure 1: Objects in the Cornell Grasping Dataset, each labelled with several grasps given as oriented 2-D rectangles.

From Figure 3: The direct regression model, five convolutional layers and three fully connected layers ending in six ... — From Figure 3: The direct regression model, five convolutional layers and three fully connected layers ending in six grasp outputs.

From Figure 4: MultiGrasp on a flying disc. Each cell of an NxN grid predicts a grasp and a probability that it is a ... — From Figure 4: MultiGrasp on a flying disc. Each cell of an NxN grid predicts a grasp and a probability that it is a true grasp; the predictions are weighted by that probability.

From Figure 5: Correct (top) and incorrect (bottom) grasps from the direct regression model. Some misses, like the ca... — From Figure 5: Correct (top) and incorrect (bottom) grasps from the direct regression model. Some misses, like the can opener, may still be viable; others, like the bowl, are not.

Abstract

We present an accurate, real-time approach to robotic grasp detection based on convolutional neural networks. Our network performs single-stage regression to graspable bounding boxes without using standard sliding window or region proposal techniques. The model outperforms state-of-the-art approaches by 14 percentage points and runs at 13 frames per second on a GPU. Our network can simultaneously perform classification so that in a single step it recognizes the object and finds a good grasp rectangle. A modification to this model predicts multiple grasps per object by using a locally constrained prediction mechanism. The locally constrained model performs significantly better, especially on objects that can be grasped in a variety of ways.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong