MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare

Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, Josef Sivic

Seminal December 2022

A Franka Panda grasps the white cleanser bottle from a cluttered table, 4x speed.

TL;DR A render-and-compare pose estimator trained once on 2 million synthetic images of over 20K objects, then applied without retraining to novel objects given only their CAD model and a 2D detection. Its RGB-D refiner adds +41.0 mean AR over its coarse estimates across the 7 BOP datasets, competitive with methods trained on the test objects.

What they do

MegaPose estimates the 6D pose of objects never seen during training. At test time it needs only the object’s CAD model and a 2D region of interest, and works from RGB or RGB-D. It keeps the detect / coarse / refine pipeline of CosyPose, but no network weights are tied to particular objects.

How it works

Refiner. A ResNet-34 takes the observed crop and renderings at the current pose and predicts a pose update. To give the network the object’s shape and anchor point, it renders 4 views whose image centers all project onto the anchor point, with depth and surface-normal channels.
Coarse model. Instead of regressing a pose, a classifier scores whether each of 520 rendered pose hypotheses falls within the refiner’s basin of attraction. Several poses can score as correct, so object symmetries need not be known.
Data. 2 million BlenderProc images of over 20K ShapeNet and Google Scanned Objects models dropped on a plane, with randomized materials, lighting and cameras.

Results

Across the 7 BOP datasets, the RGB-D refiner adds +41.0 mean AR over the coarse estimates. Started from CosyPose’s coarse poses, the RGB-only refiner beats CosyPose’s own object-specific refiner on average.
With PPF and SIFT hypotheses it scores +10.7 AR over Zephyr on YCB-V, and 59.7 vs 55.7 averaged over YCB-V and LM-O.
Ablations: 1 to 4 rendered views raises BOP5 AR from 52.0 to 61.7; 110 to 21K training objects raises it from 47.9 to 61.7 and ModelNet ADD(0.1d) from 28.7 to 96.1.
One refiner iteration takes about 50 ms, so tracking runs at 20 Hz. In a qualitative test, a Franka Panda grasped YCB objects using poses from a single RGB image.

Limitations

Most failures come from coarse estimates outside the refiner’s basin of attraction: texture-less objects that look alike from different views (a bowl), or poses that only fine details tell apart (scissors). The coarse stage takes seconds per object. A CAD model with the wrong scale gives wrong depth from RGB. Detecting novel objects is left to other work.

Videos

A second grasp among distractor objects, 4x speed.

Tracking a hand-held mustard bottle: predicted contour (left) beside the raw frame (right).

Figures

From Figure 1: (a) Training on millions of synthetic scenes with thousands of objects; (b) inference on a novel objec... — From Figure 1: (a) Training on millions of synthetic scenes with thousands of objects; (b) inference on a novel object from its CAD model and a region of interest; (c) visually guided robotic manipulation.

From Figure 2: (a) The coarse network renders the object in many pose hypotheses and classifies which rendering best ... — From Figure 2: (a) The coarse network renders the object in many pose hypotheses and classifies which rendering best matches the observed crop. (b) The refiner renders the current estimate (blue axes) plus 3 extra viewpoints (green axes) whose camera z-axis passes through the anchor point, and predicts an updated pose from the observed and rendered images.

From Figure 3: Coarse estimate (left of each pair) and the result after 5 refiner iterations (right) on novel objects... — From Figure 3: Coarse estimate (left of each pair) and the result after 5 refiner iterations (right) on novel objects from YCB-V, LM-O, HB and T-LESS, under severe occlusion; top right is a texture-less industrial plug.

Abstract

We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong