Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, Dieter Fox

Seminal November 2017

TL;DR PoseCNN labels each pixel, lets pixels vote for each object's 2D center to recover its 3D translation, and regresses rotation as a quaternion, with a loss that tolerates symmetric objects. With ICP refinement it reaches 78.0% mean accuracy on OccludedLINEMOD, and it comes with the new YCB-Video dataset of 133,827 annotated frames.

What they do

PoseCNN estimates the 6D pose of known objects in cluttered scenes, where occlusion and symmetric objects break template matching and 3D-coordinate regression. It splits the problem into semantic labeling, 3D translation and 3D rotation, computed by branches on a shared VGG16-initialized backbone. The authors also release YCB-Video: 21 YCB objects in 92 RGB-D videos with 133,827 frames, annotated by hand only in each video’s first frame and propagated by depth tracking and global optimization.

How it works

  • Translation by voting. Each pixel predicts a unit vector toward its object’s 2D center and the center’s depth. A Hough voting layer accumulates votes along these rays, so the center is found even when it is occluded. Depth is the mean over the inlier pixels, and the pinhole model gives the 3D translation.
  • Rotation. The inliers define a bounding box; RoI pooling crops features inside it and three fully connected layers regress one quaternion per class.
  • ShapeMatch-Loss. Instead of comparing corresponding model points, it measures each rotated point’s distance to the closest point on the ground-truth model, so rotations that are equivalent under the object’s symmetry are not penalized. No symmetry labels are needed.
  • Optional depth refinement. ICP with a point-to-plane residual, run from several perturbed starting poses.

Results

  • YCB-Video (area under the accuracy-threshold curve, up to 10 cm): color-only PoseCNN scores 53.7 ADD / 75.9 ADD-S, against 15.1 / 29.8 for a 3D-coordinate regression network with RANSAC. With ICP: 79.3 / 93.0 against 74.5 / 90.1.
  • OccludedLINEMOD with ICP: 78.0% mean accuracy, against 76.7% for the best prior RGB-D method. The largest gains are on the symmetric Eggbox (72.2% vs at most 65.5%) and Glue.
  • With the plain point-matching loss, rotation errors on the symmetric wood block and large clamp spread from 0 to 180 degrees; with ShapeMatch-Loss they concentrate at the symmetric equivalents.

Limitations

Color alone gives only 24.9% mean accuracy on OccludedLINEMOD, where thresholds are usually under 2 cm, so precise poses need depth and ICP. The method assumes a 3D model of every object. The two clamps look the same and get confused; the extra-large clamp scores 28.7 ADD even with ICP. The authors note that ShapeMatch-Loss can get stuck in local minima, as ICP does.

Figures

From Figure 1: From one RGB image, PoseCNN predicts semantic labels, 3D translation and 3D rotation, which combine in...
From Figure 1: From one RGB image, PoseCNN predicts semantic labels, 3D translation and 3D rotation, which combine into a 6D pose per object.
From Figure 2: Architecture of PoseCNN. A shared convolutional backbone feeds a labeling branch, a branch for center ...
From Figure 2: Architecture of PoseCNN. A shared convolutional backbone feeds a labeling branch, a branch for center direction and distance followed by Hough voting, and a RoI-pooled branch that regresses a quaternion per class.
From Figure 9: Results on YCB-Video and OccludedLINEMOD: input images, predicted labels and centers, poses from color...
From Figure 9: Results on YCB-Video and OccludedLINEMOD: input images, predicted labels and centers, poses from color only, and poses after ICP refinement.
Abstract

Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at https://rse-lab.cs.washington.edu/projects/posecnn/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.