Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation

He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, Leonidas J. Guibas

Seminal January 2019

TL;DR A CNN predicts, for each pixel of an unseen object, its coordinates in a normalized space shared by the whole category (NOCS), and aligning these with the depth map gives the object's metric 6D pose and size. On real test scenes it reaches 76.4% mAP at 3D IoU 50 and 23.1% at (10°, 5 cm), against 43.8% and 0.8% for a Mask R-CNN + ICP baseline.

What they do

Instance-level 6D pose methods need the exact CAD model of each object; category-level 3D detectors give boxes but not precise orientation. This paper estimates the full 6D pose and 3D size of unseen objects from known categories (bottle, bowl, camera, can, laptop, mug) in an RGB-D image, with no CAD model at test time.

How it works

NOCS. Every instance of a category is scaled so its bounding-box diagonal is 1, centered and consistently oriented inside a unit cube. A NOCS map gives, per pixel, the point’s coordinates in this cube.
Network. Mask R-CNN (ResNet50 + FPN) gets three extra heads that predict the x, y, z NOCS maps from RGB only. Classifying each coordinate into 32 bins works better than regression. A symmetry-aware loss takes the minimum over up to 6 rotations about a category’s symmetry axis.
Pose fitting. The masked depth points and the NOCS points are aligned with the Umeyama algorithm inside RANSAC, which gives scale, rotation and translation.
Data. CAMERA renders ShapeNet objects onto detected planes in 31 real tabletop scenes (300K images). A real dataset adds 8K RGB-D frames of 18 scenes with 42 scanned instances.

Results

On real test scenes (REAL275): 76.4% mAP at 3D IoU 50, 10.2% at (5°, 5 cm) and 23.1% at (10°, 5 cm); the Mask R-CNN + ICP baseline gets 43.8%, 0.8% and 0.8%. On synthetic CAMERA25: 83.9% and 40.9% (5°, 5 cm).
Without the symmetry loss, the regression network’s (5°, 5 cm) mAP on REAL275 falls from 8.1% to 1.3%. Non-context-aware compositing lowers (10°, 5 cm) mAP from 23.4% to 19.3%.
On OccludedLINEMOD: 13.9% mAP at (5°, 5 cm) versus 1.7% for PoseCNN without refinement, and 30.2% versus 17.2% on 2D projection at 5 px.
Runs at about 4 fps (210 ms network, 34 ms alignment) on a TITAN Xp.

Limitations

Pose depends on correct region proposals and category predictions. Depth is required to lift NOCS to metric coordinates. Real-data failures include missed detections, wrong classes and inconsistent coordinate maps.

Figures

From Figure 1: From a single RGB-D image (a), the method predicts a color-coded NOCS map for each object (b), then ou... — From Figure 1: From a single RGB-D image (a), the method predicts a color-coded NOCS map for each object (b), then outputs the metric 6D pose (axes) and dimensions (red boxes) of objects never seen in training (c).

From Figure 3: The CNN predicts each object's class label, instance mask and NOCS map from the RGB image; pose fittin... — From Figure 3: The CNN predicts each object's class label, instance mask and NOCS map from the RGB image; pose fitting combines each NOCS map with the depth image to get the metric 6D pose and size.

From Figure 4: CAMERA data generation detects planes in real tabletop images and renders synthetic objects onto them,... — From Figure 4: CAMERA data generation detects planes in real tabletop images and renders synthetic objects onto them, giving full ground truth (left); example mixed-reality images (middle) and the annotated real-world dataset (right).

From Figure 12: Results on the REAL275 test set; rows show the input, ground-truth and predicted NOCS maps, and groun... — From Figure 12: Results on the REAL275 test set; rows show the input, ground-truth and predicted NOCS maps, and ground-truth and predicted 6D pose and size.

Abstract

The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image. Contrary to "instance-level" 6D pose estimation tasks, our problem assumes that no exact object CAD models are available during either training or testing time. To handle different and unseen object instances in a given category, we introduce a Normalized Object Coordinate Space (NOCS)---a shared canonical representation for all possible object instances within a category. Our region-based neural network is then trained to directly infer the correspondence from observed pixels to this shared object representation (NOCS) along with other object information such as class label and instance mask. These predictions can be combined with the depth map to jointly estimate the metric 6D pose and dimensions of multiple objects in a cluttered scene. To train our network, we present a new context-aware technique to generate large amounts of fully annotated mixed reality data. To further improve our model and evaluate its performance on real data, we also provide a fully annotated real-world dataset with large environment and instance variation. Extensive experiments demonstrate that the proposed method is able to robustly estimate the pose and size of unseen object instances in real environments while also achieving state-of-the-art performance on standard 6D pose estimation benchmarks.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong