Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

6-DOF GraspNet: Variational Grasp Generation for Object Manipulation

Arsalan Mousavian, Clemens Eppner, Dieter Fox

Seminal May 2019

TL;DR A variational autoencoder samples diverse 6-DOF grasps from a partial object point cloud, and a learned evaluator scores them and refines them by gradient ascent. Trained only on simulated grasps, it picks up 17 real objects with 88% success versus 47% for GPD.

What they do

To pick up an unknown object seen from a single depth view, a robot has to choose a full 6-DOF gripper pose, and it needs many candidates because some will be unreachable or in collision. 6-DOF GraspNet generates a diverse set of parallel-jaw grasps directly from the object’s partial point cloud, then scores and improves them with a second network. Both are trained only on simulated grasps and run on a real robot with no extra steps.

How it works

  • Training data. FleX simulates a free-floating gripper on 206 ShapeNet objects (boxes, cylinders, bowls, bottles, mugs): 7.07M simulated grasps, 2.1M of them successful.
  • Grasp sampler. A PointNet++ conditional VAE with a 2-D latent space, trained on successful grasps only. At test time, sampling z from N(0, I) gives different grasps for the same point cloud.
  • Grasp evaluator. A PointNet classifier sees the object point cloud merged with a point cloud of the gripper at the grasp pose and predicts success. Hard negatives are perturbed positives that collide with or miss the object.
  • Refinement. The gradient of the success score with respect to grasp translation and Euler angles moves each grasp, at most 1 cm per step; grasps below a threshold are discarded.

Results

  • Real robot (Franka Panda, wrist-mounted RealSense D415, 17 objects, 51 trials): 88% success versus 47% for GPD. On mugs it is 86% versus 6%, because GPD generates no grasps on the rim.
  • In simulation, refinement raises both success and coverage, plateauing after 10 iterations, and the learned sampler beats a surface-normal sampler on both.
  • A forward pass takes 0.04 s and each refinement iteration 0.3 s for 200 samples on a Titan XP.

Limitations

Grasps are generated for one segmented object; reachability and clutter are handled afterwards by a motion planner filtering the samples. The simulator uses a free-floating gripper without gravity and fixed friction and density. Some real failures came from grasps with a finger tangent to an object face, where small execution errors turn the grasp into a push.

Figures

From Figure 1: The Franka Panda used in the experiments, picking up unknown objects with generated grasps.
From Figure 1: The Franka Panda used in the experiments, picking up unknown objects with generated grasps.
From Figure 3: (left) Grasps are expressed relative to the center of the object point cloud, with axes parallel to th...
From Figure 3: (left) Grasps are expressed relative to the center of the object point cloud, with axes parallel to the camera. (right) The sampler proposes grasps from the object point cloud, the evaluator scores them, and grasps are refined iteratively using the evaluator's gradient.
From Figure 4: During training, the encoder maps each grasp to a latent z pushed toward a normal distribution; the de...
From Figure 4: During training, the encoder maps each grasp to a latent z pushed toward a normal distribution; the decoder reconstructs 6D grasps from the point cloud and z.
From Figure 11: Success-coverage curves for the learned VAE sampler versus a geometric surface-normal sampler, both f...
From Figure 11: Success-coverage curves for the learned VAE sampler versus a geometric surface-normal sampler, both followed by the evaluator (AUC 0.18 vs 0.07).
Abstract

Generating grasp poses is a crucial component for any robot object manipulation task. In this work, we formulate the problem of grasp generation as sampling a set of grasps using a variational autoencoder and assess and refine the sampled grasps using a grasp evaluator model. Both Grasp Sampler and Grasp Refinement networks take 3D point clouds observed by a depth camera as input. We evaluate our approach in simulation and real-world robot experiments. Our approach achieves 88\% success rate on various commonly used objects with diverse appearances, scales, and weights. Our model is trained purely in simulation and works in the real world without any extra steps. The video of our experiments can be found at: https://research.nvidia.com/publication/2019-10_6-DOF-GraspNet\%3A-Variational

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.