PointNetGPD: Detecting Grasp Configurations from Point Sets

Hongzhuo Liang, Xiaojian Ma, Shuang Li, Michael Görner, Song Tang, Bin Fang, Fuchun Sun, Jianwei Zhang

Seminal September 2018

TL;DR PointNetGPD scores parallel-jaw grasp candidates by running a PointNet classifier on the raw points inside the gripper, trained on 350k real point clouds and grasps over YCB objects. It beats the GPD baseline at grasp-quality classification (84.75% vs 79.96% on single-view clouds), and on a real UR5 it clears more of each cluttered pile (completion 95-100% vs 81.5-95%).

What they do

GPD and similar grasp evaluators project the point cloud into hand-crafted 2D images for a CNN. PointNetGPD instead gives the raw 3D points inside a candidate gripper’s closing area to a PointNet-style classifier, which the authors find holds up on sparse single-view clouds. They also build a training set of 350k real point clouds and grasps over 47 YCB objects. Each grasp gets a continuous quality score instead of a binary label.

How it works

Dataset. Grasps are sampled on YCB meshes (two random surface contacts plus an approach angle), collision-checked with a gripper model and aligned to the real point clouds by ICP. The score is 1/γ, where γ is the smallest friction coefficient (searching up from 0.4) that makes the grasp antipodal, plus 0.01 times a frictionless grasp-wrench-space score.
Network. Points inside the gripper are moved into a local gripper frame, resampled to 1000 points and sorted into 2 or 3 quality classes by a 1.6M-parameter PointNet.
Candidates. A modified GPG sampler proposes grasps from the cloud. It drops points near the support surface and backs colliding grasps off along their approach direction.

Results

Classification: the 2-class model reaches 84.75% on single-view and 91.81% on full clouds. The best GPD variant reaches 79.96% and 84.29% with 3.64M parameters.
Isolated objects (UR5, Robotiq gripper in pinch mode, one Kinect2, 22 YCB objects with 11 unseen, 10 trials each): over the ten objects where results differed, Table II reports 81% (2-class) and 82% (3-class) average success vs 49% for GPD. The GPD values listed in that table actually average 58%.
Clutter removal, two object sets, 5 rounds each: best-class grasps from the 3-class model succeed 89.33%/66.20% with 100%/95% completion, vs 84.83%/61.13% and 95%/81.5% for GPD. Second-class grasps succeed only 52.10%/43.75%.

Limitations

With one fixed depth view, occlusion causes failures in clutter, and the model sometimes grasps several objects at once as if they were one. Candidate generation is a separate heuristic outside the network. No model ever grasped the medium clamp.

Videos

Figures

From Figure 1: The pipeline. The depth map becomes a point cloud, grasp candidates are sampled, the points inside eac... — From Figure 1: The pipeline. The depth map becomes a point cloud, grasp candidates are sampled, the points inside each gripper are cropped into local coordinates and scored by the network, and the best grasp is executed. Training uses a grasp dataset built on YCB objects.

From Figure 4: The local gripper frame and the points inside the gripper closing area (magenta), which together repre... — From Figure 4: The local gripper frame and the points inside the gripper closing area (magenta), which together represent a grasp.

From Figure 3: The PointNet-based quality network. The points inside the gripper, in gripper coordinates, pass throug... — From Figure 3: The PointNet-based quality network. The points inside the gripper, in gripper coordinates, pass through spatial transforms and feature extraction, and a global feature classifies the grasp's quality level.

From Figure 5: Test accuracy over training on single-view (top) and full (bottom) point clouds. Every model does bett... — From Figure 5: Test accuracy over training on single-view (top) and full (bottom) point clouds. Every model does better with the full cloud, and PointNetGPD beats the GPD baselines on both.

Abstract

In this paper, we propose an end-to-end grasp evaluation model to address the challenging problem of localizing robot grasp configurations directly from the point cloud. Compared to recent grasp evaluation metrics that are based on handcrafted depth features and a convolutional neural network (CNN), our proposed PointNetGPD is lightweight and can directly process the 3D point cloud that locates within the gripper for grasp evaluation. Taking the raw point cloud as input, our proposed grasp evaluation network can capture the complex geometric structure of the contact area between the gripper and the object even if the point cloud is very sparse. To further improve our proposed model, we generate a larger-scale grasp dataset with 350k real point cloud and grasps with the YCB object set for training. The performance of the proposed model is quantitatively measured both in simulation and on robotic hardware. Experiments on object grasping and clutter removal show that our proposed model generalizes well to novel objects and outperforms state-of-the-art methods. Code and video are available at \href{https://lianghongzhuo.github.io/PointNetGPD}{https://lianghongzhuo.github.io/PointNetGPD}

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong