Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes

Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter Fox

★ Must read March 2021

TL;DR A single PointNet++ network treats each observed depth point as a possible grasp contact and predicts a full 6-DoF parallel-jaw grasp there, so it only has to learn 4 DoF. Trained on simulated grasps only, it picks unseen objects from cluttered scenes on a Franka with 90.2% success, versus 80.4% for the prior best pipeline.

What they do

Earlier 6-DoF grasping pipelines generate grasps from a segmented object point cloud and then filter collisions with a second network, so they depend on accurate masks and ignore the surrounding scene. Contact-GraspNet predicts grasps in one pass from the scene point cloud of a depth image. Instance segmentation is only used afterwards, to assign grasps to a target object.

How it works

  • Contact representation. Each grasp is rooted at a visible contact point in the point cloud. Per point, the network predicts a contact score, the approach and baseline directions (orthonormalized with Gram-Schmidt) and the grasp width as one of 10 bins.
  • Data. ACRONYM: 8872 ShapeNet meshes with 17.7 million simulated grasps, placed in 10000 table-top scenes of 8-12 objects. Colliding grasps are removed, and rendered points within 5 mm of a mesh contact are labeled positive.
  • Network and loss. A PointNet++ U-net takes 20000 points and predicts at 2048 of them. The loss combines top-k cross entropy on contacts, a confidence-weighted distance from predicted gripper points to the closest ground-truth grasp, and a bin-weighted width loss. Training takes about 40 h on one V100.
  • Inference. 0.28 s for a full scene, about 0.19 s for a region around a target.

Results

  • Franka Panda, 9 cluttered scenes with 51 unseen objects, at most two tries per object: 90.20% success, 84.31% on the first try, 59 attempts in total. 6-DOF GraspNet + CollisionNet reaches 80.39%, 68.63% and 67 attempts; 6-DOF GraspNet alone 62.7%.
  • Simulation ablations: without bin weighting, width predictions collapse to narrow grasps; the distance loss raises success of high-confidence grasps; local regions raise coverage over full-scene prediction; training on a 5-category dataset does not generalize to other objects.

Limitations

Only parallel-jaw grasps with full surface contact, by design (no grasps through a mug handle), and at least one grasp contact must be visible to the camera. Confidence drops on thick objects that need nearly the maximum width and on small objects. Targets in the robot study were selected manually.

Figures

From Figure 1: Contact-GraspNet predicts diverse, stable grasps in cluttered scenes while avoiding collisions.
From Figure 1: Contact-GraspNet predicts diverse, stable grasps in cluttered scenes while avoiding collisions.
From Figure 3: The grasp representation. A grasp is rooted at an observed contact point c; the approach vector a and ...
From Figure 3: The grasp representation. A grasp is rooted at an observed contact point c; the approach vector a and baseline vector b give its 3-DoF rotation, w is the grasp width and d the distance from baseline to gripper base. The five pink gripper points are used in the pose loss.
From Figure 4: Inference pipeline. Unknown objects are segmented from the RGB-D image, Contact-GraspNet predicts gras...
From Figure 4: Inference pipeline. Unknown objects are segmented from the RGB-D image, Contact-GraspNet predicts grasps on the full scene point cloud or a region around a target, and grasps are assigned to segments by their contact points. Right: the predicted grasps, with the most confident grasp per segment in bold.
From Figure 7: Successful grasp contacts are still found on the drill despite severe under-segmentation.
From Figure 7: Successful grasp contacts are still found on the drill despite severe under-segmentation.
Abstract

Grasping unseen objects in unconstrained, cluttered environments is an essential skill for autonomous robotic manipulation. Despite recent progress in full 6-DoF grasp learning, existing approaches often consist of complex sequential pipelines that possess several potential failure points and run-times unsuitable for closed-loop grasping. Therefore, we propose an end-to-end network that efficiently generates a distribution of 6-DoF parallel-jaw grasps directly from a depth recording of a scene. Our novel grasp representation treats 3D points of the recorded point cloud as potential grasp contacts. By rooting the full 6-DoF grasp pose and width in the observed point cloud, we can reduce the dimensionality of our grasp representation to 4-DoF which greatly facilitates the learning process. Our class-agnostic approach is trained on 17 million simulated grasps and generalizes well to real world sensor data. In a robotic grasping study of unseen objects in structured clutter we achieve over 90% success rate, cutting the failure rate in half compared to a recent state-of-the-art method.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.