Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching

Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R. Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, Nima Fazeli, Ferran Alet, Nikhil Chavan Dafle, Rachel Holladay, Isabella Morona, Prem Qu Nair, Druck Green, Ian Taylor, Weber Liu, Thomas Funkhouser, Alberto Rodriguez

Seminal October 2017

TL;DR A pick-and-place system that first grasps whatever its pixel-wise affordance maps rank highest across four suction and parallel-jaw primitives, then identifies the held object by matching it to product images, so novel objects need no retraining. It won the 2017 Amazon Robotics Challenge stowing task, stowing all 20 objects with 100% recognition accuracy.

What they do

A pick-and-place system for objects it has never seen, given only their product images. It reverses the usual recognize-then-grasp order: it picks an object without knowing what it is, lifts it away from the clutter, then recognizes it. The system was part of the MIT-Princeton entry that won the stowing task of the 2017 Amazon Robotics Challenge (ARC).

How it works

Four primitives. A gripper with parallel jaws and a retractable suction cup can suction down, suction from the side, grasp down, or flush-grasp an object lying against the bin wall.
Dense affordances. One FCN scores every pixel of each RGB-D view for suction. A second scores top-down grasps on a heightmap, rotated to cover 16 gripper angles. Both are built on ResNet-101 and trained on 1837 hand-labeled images. A planner executes the highest-scoring proposal.
Recognition by matching. A two-stream network learns to map camera images of the held object onto the ImageNet features of a frozen product-image stream, so novel objects are identified by nearest neighbor. A first stage decides whether the object is known or novel and routes it to the network trained for that case.

Results

Top-1 suction precision is 92.4% (Dex-Net 69.3%, heuristic baseline 35.2%); top-1 grasp precision is 96.7% (Dex-Net 80.4%, baseline 92.5%).
On a 1-vs-20 recognition benchmark, the two-stage system scores 88.6% on mixed objects (93.6% known, 77.5% novel), vs 74.2% for a Siamese network.
ARC 2017 stow: 58.3% suction and 75% grasp pick success and 100% recognition, with all 20 objects stowed. At the competition, parallel-jaw grasps came from the heuristic baseline because the grasp network was not yet stable.

Limitations

It only picks objects that a single primitive can reach directly; it does not push or clear other objects first. Grasps run open loop, so small affordance errors lead to failed or marginal grasps. Picking a specific target object needs state tracking, which loses track when several objects move at once. In the finals, a camera disconnection also forced a hard reset.

Videos

Figures

From Figure 1: The system computes pixel-wise grasp affordances over a cluttered bin (a), grasps a towel and holds it... — From Figure 1: The system computes pixel-wise grasp affordances over a cluttered bin (a), grasps a towel and holds it away from the clutter, then recognizes it by matching its observed images (b) to a product image.

From Figure 5: One FCN infers pixel-wise suction affordances on each RGB-D view (top). Another scores horizontal gras... — From Figure 5: One FCN infers pixel-wise suction affordances on each RGB-D view (top). Another scores horizontal grasps on an orthographic heightmap rotated to 16 angles (bottom). The proposal with the highest affordance is executed.

From Figure 6: A two-stream network maps product images and observed images into a common feature space. At test time... — From Figure 6: A two-stream network maps product images and observed images into a common feature space. At test time, an observed object is recognized by its nearest product image, for known and novel objects alike.

From Figure 8: Dex-Net top-1 predictions (red) vs this method's (green) for suction (left) and grasping (right). This... — From Figure 8: Dex-Net top-1 predictions (red) vs this method's (green) for suction (left) and grasping (right). This method picks nearer the center of mass, avoids a porous mesh bag and is less affected by noisy depth.

Abstract

This paper presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses a category-agnostic affordance prediction algorithm to select and execute among four different grasping primitive behaviors. It then recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data. Exhaustive experimental results demonstrate that our multi-affordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT-Princeton Team system that took 1st place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at http://arc.cs.princeton.edu

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong