Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network

Sulabh Kumra, Shirin Joshi, Ferat Sahin

Seminal September 2019

TL;DR GR-ConvNet, a 1.9M-parameter residual network, turns an RGB-D image into per-pixel grasp quality, angle and width maps in about 20 ms. It scores 97.7% on Cornell and 94.6% on Jacquard, and a Baxter arm picks 95.4% of household and 93% of adversarial objects.

What they do

GR-ConvNet predicts antipodal parallel-jaw grasps for unknown objects from an n-channel image (depth, RGB or RGB-D). Instead of scoring sampled candidates, it outputs a grasp at every pixel in one forward pass. A ROS control module executes the best one on a robot arm.

How it works

  • Grasp representation. Each pixel gets a quality score in [0, 1], a gripper angle in [-π/2, π/2] and a gripper width. The grasp pose in the robot frame is a 3D position plus a rotation about the z-axis, obtained through camera intrinsics and hand-eye calibration.
  • Network. A 224x224 input passes through three convolutional layers, five residual blocks and transposed convolutions back to full resolution. It outputs four images: quality, cos 2θ, sin 2θ and width. The network has 1,900,900 parameters, small enough for closed-loop control at up to 50 Hz.
  • Training. Smooth L1 (Huber) loss against ground-truth grasp images, Adam with learning rate 1e-3. Cornell is augmented with random crops, zooms and rotations to about 51k grasps; Jacquard (1.1M grasps) is used as is.

Results

  • Cornell: 97.7% image-wise and 96.6% object-wise with RGB-D at 20 ms. This ties the accuracy of FCGN (ResNet-101), which takes 117 ms. Depth-only gets 93.2% / 94.3%.
  • Jacquard: 94.6% with RGB-D, ahead of FCGN’s 91.8%.
  • Baxter with a parallel gripper and a RealSense D435: 334/350 grasps (95.4%) on 35 household objects and 93/100 (93%) on 10 3D-printed adversarial objects.
  • Clutter: 93.5% grasp success over 10 runs with 60 unseen objects, although training used only isolated objects.

Limitations

The model gave a poor grasp for a transparent bottle, which the authors attribute to bad depth data. Other failures were very low grasp scores, objects slipping from the closed gripper, and collisions with nearby objects. Grasps are planar (rotation about z only), and the system only handles parallel-jaw grippers.

Videos

Full video (YouTube)

Figures

From Figure 1: System overview. The inference module predicts grasp poses for objects in the camera's view; the contr...
From Figure 1: System overview. The inference module predicts grasp poses for objects in the camera's view; the control module uses them to plan and execute robot trajectories for antipodal grasps.
From Figure 2: The GR-ConvNet architecture: convolutional layers, five residual blocks and transposed convolutions th...
From Figure 2: The GR-ConvNet architecture: convolutional layers, five residual blocks and transposed convolutions that output quality, angle and width images.
From Figure 4: Predicted quality, angle and width maps and the resulting grasp rectangles for (a) unseen Cornell obje...
From Figure 4: Predicted quality, angle and width maps and the resulting grasp rectangles for (a) unseen Cornell objects, (b) unseen Jacquard objects, (c) a single household object, (d) multiple objects, and (e) a poor grasp on a transparent object.
From Figure 5: The robot grasping adversarial (top) and household (bottom) objects in clutter: (a) predicted grasp po...
From Figure 5: The robot grasping adversarial (top) and household (bottom) objects in clutter: (a) predicted grasp pose, (b) approach, (c) grasp, (d) retract.
Abstract

In this paper, we present a modular robotic system to tackle the problem of generating and performing antipodal robotic grasps for unknown objects from n-channel image of the scene. We propose a novel Generative Residual Convolutional Neural Network (GR-ConvNet) model that can generate robust antipodal grasps from n-channel input at real-time speeds (~20ms). We evaluate the proposed model architecture on standard datasets and a diverse set of household objects. We achieved state-of-the-art accuracy of 97.7% and 94.6% on Cornell and Jacquard grasping datasets respectively. We also demonstrate a grasp success rate of 95.4% and 93% on household and adversarial objects respectively using a 7 DoF robotic arm.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.