Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg

★ Must read March 2017

TL;DR Train a CNN to score parallel-jaw grasps from depth images using only synthetic data, 6.7 million rendered point clouds labeled with a robust analytic grasp metric. On an ABB YuMi it plans grasps in 0.8 s with 93% success on known adversarial objects and reaches 99% precision on 40 novel household objects.

What they do

Learning grasps from human labels or physical trials takes tedious labeling or months of robot time. Dex-Net 2.0 instead renders 6.7 million synthetic depth images of 1,500 3D models, labels parallel-jaw grasps with a robust analytic metric, and trains a Grasp Quality CNN (GQ-CNN) that predicts grasp success directly from a depth image.

How it works

Dataset. Meshes from Dex-Net 1.0 (3DNet and KIT) are placed in stable poses with random planar pose, camera pose and friction. Up to 100 antipodal grasps per object are scored with robust epsilon quality under pose and friction uncertainty. A grasp is labeled positive if it clears a threshold and is collision-free (about 21% of the data).
GQ-CNN. Input is a depth image centered on the grasp and rotated to its axis, plus the gripper’s depth. Depth noise (multiplicative Gamma plus Gaussian Process) is resampled during training.
Planning. Sample antipodal candidates from depth-image gradients, score each with the GQ-CNN, and execute the most robust grasp that is reachable and does not hit the table.

Results

Known objects (8 adversarial 3D-printed parts, 80 trials): the model finetuned on synthetic images of these parts reaches 93% success and 94% precision in 0.8 s. Point-cloud registration reaches 95% but takes 2.6 s. Pretraining on a 150-model subset instead of the full dataset gives 85%.
Novel objects (10 household items, 50 trials): 80% success and 100% precision (29 grasps rated robust, no false positives), versus 52% for registration and 60% for an image-based heuristic.
40 novel objects, some articulated or deformable, with cross-entropy-method planning: 94% success and 99% precision (68 of 69), at 2.5 s per plan.
Limitations
The method only handles singulated rigid objects on a table, seen from one overhead depth view, with planar grasps. The order-fulfillment demo needed a separate pushing policy to break up piles. Failures came from the depth sensor missing thin parts and from grasps in narrow openings that collide with the object but were rated robust.

Videos

Figures

From Figure 1: Dex-Net 2.0 overview. A GQ-CNN trained offline on 6.7 million synthetic point clouds, grasps and robus... — From Figure 1: Dex-Net 2.0 overview. A GQ-CNN trained offline on 6.7 million synthetic point clouds, grasps and robust grasp metrics ranks several hundred antipodal grasp candidates from a depth image, and the ABB YuMi executes the most robust one.

From Figure 3: Dataset generation. Grasps sampled on 1,500 object meshes are scored with robust analytic metrics, dep... — From Figure 3: Dataset generation. Grasps sampled on 1,500 object meshes are scored with robust analytic metrics, depth images are rendered for each stable pose with random object and camera poses, and each grasp becomes a 32x32 image centered and aligned on the grasp.

From Figure 4: GQ-CNN architecture. A depth image aligned to the grasp center and axis, plus the gripper depth from t... — From Figure 4: GQ-CNN architecture. A depth image aligned to the grasp center and axis, plus the gripper depth from the camera, pass through four convolutional and three fully connected layers to a probability of grasp success. Right: the learned first-layer filters.

From Figure 7: ROC curves for classifying robust grasps on held-out synthetic images of the adversarial training obje... — From Figure 7: ROC curves for classifying robust grasps on held-out synthetic images of the adversarial training objects. The GQ-CNN variants outperform a random forest (ML-RF) and an SVM (ML-SVM).

Abstract

To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8sec with a success rate of 93% on eight known objects with adversarial geometry and is 3x faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong