Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

DeepIM: Deep Iterative Matching for 6D Pose Estimation

Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, Dieter Fox

Seminal March 2018

TL;DR DeepIM refines an initial 6D object pose from one RGB image by repeatedly rendering the object's 3D model and predicting the pose correction that aligns the rendering with the image. Starting from PoseCNN, it raises LINEMOD 6D Pose accuracy from 62.7% to 88.6% and also refines objects never seen in training.

What they do

RGB-only 6D pose estimators that classify or regress the pose directly are not very accurate, because small errors turn into pose mismatches. DeepIM is a refinement network. Given an image, the object’s 3D model and an initial pose (for example from PoseCNN), it renders the model at that pose and predicts the relative SE(3) transform that matches the rendering to the image. It then repeats the process from the updated pose.

How it works

  • Zoomed input. The observed image, the rendered image and both masks are cropped around the object with an enlarged box, upsampled to 480x640 and stacked into an 8-channel input to a pretrained FlowNetSimple backbone. Optical flow and mask heads are used only during training.
  • Disentangled update. Rotation is about the object’s centre, with axes parallel to the camera frame. Translation is a pixel shift in x and y plus a log scale change. The update needs no object size, model frame or camera intrinsics.
  • Point-matching loss. The loss is the average L1 distance between 3,000 model points under the predicted and true poses. It replaces separate rotation and translation losses.
  • Iterative training. Training poses are ground truth plus noise (rotation within 45°). The network’s own refined poses become further training samples.

Results

  • LINEMOD, starting from PoseCNN: 5°/5 cm accuracy goes from 19.4% to 85.2%, 6D Pose (ADD) from 62.7% to 88.6% and 2D projection from 70.2% to 97.5%. Starting from a weaker Faster R-CNN baseline (33.1% ADD) still reaches 86.9%.
  • Occlusion LINEMOD: ADD goes from 26.9% to 55.5%. YCB-Video: AUC of ADD(-S) goes from 61.3 to 81.9 (85.9 when tracking frame to frame), against 86.6 for PoseCNN with depth-based ICP. It runs at 12 fps per object with 2 iterations on an NVIDIA 1080 Ti.
  • Ablations: if the network is trained for a single step, extra test iterations do not help. Removing the zoom-in drops 5°/5 cm accuracy from 85.2% to 38.3%.
  • On ModelNet objects and categories never seen in training, ADD rises from 9-26% to 70-97%.

Limitations

DeepIM needs a 3D model of each object and an initial pose. The failure cases are initial poses too far from the truth, a mismatch between model and image (bad lighting or an inaccurate model), featureless or heavily occluded views, unusual shapes, and tiny key parts. Rotationally symmetric objects need a special loss with a manually calibrated axis. Without depth it still trails depth-based methods on YCB-Video (ADD-S 88.1 vs 93.0 for DenseFusion).

Figures

From Figure 1: The iterative loop. The 3D model is rendered at the current pose, the network compares the rendering w...
From Figure 1: The iterative loop. The 3D model is rendered at the current pose, the network compares the rendering with the observed image and predicts a relative pose change, and the refined pose feeds the next iteration.
From Figure 3: The network. Observed and rendered images and their masks pass through a FlowNetSimple backbone; fully...
From Figure 3: The network. Observed and rendered images and their masks pass through a FlowNetSimple backbone; fully connected layers predict rotation and translation, while flow and mask branches are used only in training.
From Figure 9: Refined poses on Occlusion LINEMOD starting from PoseCNN. Red lines are the initial silhouettes, green...
From Figure 9: Refined poses on Occlusion LINEMOD starting from PoseCNN. Red lines are the initial silhouettes, green lines the refined ones.
Abstract

Estimating the 6D pose of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the observed image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using an untangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over state-of-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.