Jacquard: A Large Scale Dataset for Robotic Grasp Detection

Amaury Depierre, Emmanuel Dellandréa, Liming Chen

Seminal March 2018

TL;DR A synthetic grasp dataset: 54k RGB-D images of 11k ShapeNet objects, labeled with 1.1M grasps that succeeded in a physics simulator. An AlexNet trained on it succeeds on 78.43% of real-robot grasps, against 60.46% when trained on the hand-labeled Cornell dataset.

What they do

Image-based grasp detectors need lots of labeled data; the main public set, Cornell, has 885 hand-labeled images. Jacquard generates the labels in simulation instead: CAD models from ShapeNetSem are dropped onto a plane, rendered as RGB-D, and annotated with every grasp that succeeds in a physics simulator. The paper also proposes judging predictions by simulated grasp trials (SGT) rather than by overlap with annotated rectangles.

How it works

Scenes. Each object is rescaled so its longest side is 8-90 cm, given a mass of 80-900 g, and dropped at a random pose; up to five scenes per object.
Rendering. Blender renders RGB and depth. For realistic noisy depth, two extra images with a projected pattern go through an off-the-shelf stereo algorithm.
Labels. In pyBullet, thousands of random candidates (sampled more densely along aligned image edges) are tried with a 2 cm jaw; successes are retried with jaw sizes of 1-6 cm. A grasp counts if the object is lifted, moved and dropped at a target.
SGT. A prediction is scored by rebuilding the scene and executing it in the simulator, so valid grasps that were never annotated still count.

Results

An ImageNet-pretrained AlexNet trained on Jacquard reaches 81.92% on Cornell (rectangle metric), close to the 86.88% of training on Cornell itself.
Trained on Cornell and tested on Jacquard, it drops to 54.28% (rectangle) and 42.76% (SGT), against 74.21% and 72.42% when trained on Jacquard.
On a Fanuc M-20iA with 28 real objects, the Jacquard-trained network succeeds on 78.43% of grasps versus 60.46% for Cornell. Jacquard failures were mostly unstable grasps dropped in transit; Cornell failures were mostly badly placed rectangles.

Limitations

Scenes hold a single object on a plane; multi-object scenes are left to future work. Grasps are planar rectangles with the approach vector set by the surface normal. All tests use one network, AlexNet, and the paper does not say how many real grasps were attempted.

Figures

From Figure 1: Jacquard images of varied objects, each with many labeled grasps drawn as 2D rectangles; the darker si... — From Figure 1: Jacquard images of varied objects, each with many labeled grasps drawn as 2D rectangles; the darker sides mark the jaws.

From Figure 4: grasps the rectangle metric misjudges. Predictions in yellow and green, ground truth in red and purple... — From Figure 4: grasps the rectangle metric misjudges. Predictions in yellow and green, ground truth in red and purple; top row false positives, bottom row false negatives.

From Figure 6: grasp predictions on the real setup from the network trained on Cornell (top row) and on Jacquard (bot... — From Figure 6: grasp predictions on the real setup from the network trained on Cornell (top row) and on Jacquard (bottom row).

Abstract

Grasping skill is a major ability that a wide number of real-life applications require for robotisation. State-of-the-art robotic grasping methods perform prediction of object grasp locations based on deep neural networks. However, such networks require huge amount of labeled data for training making this approach often impracticable in robotics. In this paper, we propose a method to generate a large scale synthetic dataset with ground truth, which we refer to as the Jacquard grasping dataset. Jacquard is built on a subset of ShapeNet, a large CAD models dataset, and contains both RGB-D images and annotations of successful grasping positions based on grasp attempts performed in a simulated environment. We carried out experiments using an off-the-shelf CNN, with three different evaluation metrics, including real grasping robot trials. The results show that Jacquard enables much better generalization skills than a human labeled dataset thanks to its diversity of objects and grasping positions. For the purpose of reproducible research in robotics, we are releasing along with the Jacquard dataset a web interface for researchers to evaluate the successfulness of their grasping position detections using our dataset.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong