Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter

Michel Breyer, Jen Jen Chung, Lionel Ott, Roland Siegwart, Juan Nieto

Seminal January 2021

TL;DR A 3D fully convolutional network takes a TSDF of a cluttered scene and predicts grasp quality, gripper orientation and opening width at every voxel in one 10 ms forward pass. Trained only on simulated grasps, it clears 92% of objects in real-robot clutter removal without explicit collision checking.

What they do

Earlier 6 DOF grasp detectors often reasoned on isolated objects, needing separate collision checks in clutter, or took seconds per plan. VGN maps a TSDF of the whole workspace to a dense grasp map: for every voxel, the success probability of a parallel-jaw grasp centred there, plus its orientation and opening width. One forward pass gives all grasps, with no candidate sampling.

How it works

  • Network. A 40³ TSDF of a 30 cm workspace goes through three strided 3D convolutions (16, 32, 64 filters) down to 64×5³, then three convolutions with 2× upsampling and three heads: quality, a normalized quaternion, and width.
  • Loss. Binary cross-entropy on quality, MSE on width, and a quaternion loss that takes the better of the two rotations 180° apart about the wrist axis, so the gripper’s symmetry does not give conflicting targets. Loss is computed only at labeled voxels.
  • Data. In PyBullet, 303 training meshes form “pile” or “packed” scenes; points from fused synthetic depth images are tried at 6 orientations about the surface normal, with collisions labeled as failures. About 2 million balanced grasps.
  • Detection. Smooth the quality volume, mask voxels closer to a surface than the finger depth, threshold, run non-maxima suppression and read out orientation and width.

Results

  • Simulated clutter removal with 5 unseen objects (ε = 0.9), success / % cleared: pile 62.3 / 46.4 vs. GPD 59.9 / 26.1; packed 87.6 / 80.4 vs. 73.7 / 72.8.
  • In 10-object piles, GPD has higher success (63.1 vs. 59.3) but clears 17.0% of objects to VGN’s 34.6%.
  • Planning takes 10 ms on a GTX 1080 Ti versus 1.2 s for GPD; on the robot’s CPU-only computer it takes 1.25 s.
  • On a Franka Panda with a wrist RealSense and no fine-tuning, 55 of 68 grasps succeed (80%) and 92% of objects are cleared; 9 of 13 failures are on a lint roller or hand gel.

Limitations

Simulated friction is too forgiving, so shallow grasps on cylinders are predicted with confidence and slip in reality. Each voxel gets one orientation, not a distribution. Transparent and reflective objects were excluded, other grippers are untested, and the scan trajectory is fixed and open-loop.

Figures

From Figure 1: A scan of the scene is fused into a TSDF; VGN's three heads output grasp quality, gripper orientation ...
From Figure 1: A scan of the scene is fused into a TSDF; VGN's three heads output grasp quality, gripper orientation and gripper width at every voxel, and non-maxima suppression plus filtering against the TSDF give the final grasps.
From Figure 2: Simulated
From Figure 2: Simulated "pile" (a) and "packed" (b) training scenes, the grasp frame defined by the gripper geometry (c), and the distribution of angles between gravity and the grasp z axis over the training set (d).
From Figure 4: Real-robot grasps by VGN (a-b), a typical failure where the fingers slip off a cylindrical object (c),...
From Figure 4: Real-robot grasps by VGN (a-b), a typical failure where the fingers slip off a cylindrical object (c), a side-grasp (d) and a grasp on the thin rim of a bowl (e).
Abstract

General robot grasping in clutter requires the ability to synthesize grasps that work for previously unseen objects and that are also robust to physical interactions, such as collisions with other objects in the scene. In this work, we design and train a network that predicts 6 DOF grasps from 3D scene information gathered from an on-board sensor such as a wrist-mounted depth camera. Our proposed Volumetric Grasping Network (VGN) accepts a Truncated Signed Distance Function (TSDF) representation of the scene and directly outputs the predicted grasp quality and the associated gripper orientation and opening width for each voxel in the queried 3D volume. We show that our approach can plan grasps in only 10 ms and is able to clear 92% of the objects in real-world clutter removal experiments without the need for explicit collision checking. The real-time capability opens up the possibility for closed-loop grasp planning, allowing robots to handle disturbances, recover from errors and provide increased robustness. Code is available at https://github.com/ethz-asl/vgn.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.