DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion

Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, Silvio Savarese

Seminal January 2019

TL;DR DenseFusion estimates the 6D pose of known objects from RGB-D by fusing color and point-cloud features at every pixel, letting each pixel vote for a pose, and refining it with a learned network instead of ICP. On YCB-Video, 96.8% of its poses fall under 2 cm ADD-S error, versus 93.2% for PoseCNN+ICP, and it runs about 200x faster.

What they do

RGB-D pose estimators either process color and depth separately or rely on slow post-processing such as ICP. DenseFusion encodes the image with a CNN and the depth points with a PointNet, fuses them pixel by pixel, and lets each pixel predict a pose, so occluded objects can be located from their visible part. A learned refinement module replaces ICP.

How it works

Segmentation. PoseCNN’s segmentation network masks each known object; the masked depth becomes a point cloud and the mask’s box crops the image.
Dense fusion. A ResNet-18 encoder-decoder gives a 128-d color feature per pixel; a PointNet variant with average pooling gives a 128-d feature per point. Each point is paired with the pixel it projects to, and a pooled global feature is appended to every pair.
Per-pixel voting. Each fused feature predicts a pose and a confidence. The loss is the confidence-weighted distance between model points under predicted and true pose (closest-point distance for symmetric objects), plus a term penalizing low confidence. The most confident pose is the output.
Refinement. The point cloud is moved into the frame of the current estimate, re-fused with the same image features, and a 4-layer network predicts a residual pose. Two iterations, trained after the main network converges.

Results

YCB-Video: ADD-S AUC 93.1 and 96.8% under 2 cm, against 93.0 and 93.2% for PoseCNN+ICP and 83.9 and 74.1% for PointFusion. Refinement lifts the bowl from 69.5% to 98.8% under 2 cm.
As occlusion increases, PointFusion and PoseCNN+ICP degrade sharply while the per-pixel and iterative variants drop only 2%.
0.06 s per frame versus 10.6 s for PoseCNN+ICP; 16 FPS with about 5 objects per frame.
LineMOD ADD: 94.3 (86.2 without refinement) versus 79 for SSD-6D+ICP.
On a Toyota HSR with a different camera and no finetuning, 73% of 60 grasps on 5 YCB objects succeed; the banana only 7/12.

Limitations

It needs a 3D model of every object and relies on PoseCNN’s segmentation as its first stage. The large and extra-large clamps stay the weakest objects (79.2% and 76.3% under 2 cm). In the grasping test objects were never stacked, and the authors attribute the banana failures to their plain-yellow banana differing from the dataset model.

Figures

From Figure 1: An end-to-end network estimates 6D object poses from RGB-D fast enough for a robot to grasp and manipu... — From Figure 1: An end-to-end network estimates 6D object poses from RGB-D fast enough for a robot to grasp and manipulate objects.

From Figure 2: The model segments objects in the RGB image, encodes colors and the depth point cloud into embeddings,... — From Figure 2: The model segments objects in the RGB image, encodes colors and the depth point cloud into embeddings, and fuses them at each corresponding pixel. Each pixel predicts a pose, and the predictions are voted into the final 6D pose.

From Figure 3: Iterative pose refinement. A network module predicts a pose residual from the point cloud transformed ... — From Figure 3: Iterative pose refinement. A network module predicts a pose residual from the point cloud transformed by the current estimate, and repeats.

From Figure 4: Predictions on YCB-Video from PointFusion, PoseCNN+ICP and DenseFusion with 2 refinement iterations, a... — From Figure 4: Predictions on YCB-Video from PointFusion, PoseCNN+ICP and DenseFusion with 2 refinement iterations, all using the same segmentation masks. Object point clouds are transformed by the predicted pose and projected into the image.

Abstract

A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources. Prior works either extract information from the RGB image and depth separately or use costly post-processing steps, limiting their performances in highly cluttered scenes and real-time applications. In this work, we present DenseFusion, a generic framework for estimating 6D pose of a set of known objects from RGB-D images. DenseFusion is a heterogeneous architecture that processes the two data sources individually and uses a novel dense fusion network to extract pixel-wise dense feature embedding, from which the pose is estimated. Furthermore, we integrate an end-to-end iterative pose refinement procedure that further improves the pose estimation while achieving near real-time inference. Our experiments show that our method outperforms state-of-the-art approaches in two datasets, YCB-Video and LineMOD. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong