FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects

Bowen Wen, Wei Yang, Jan Kautz, Stan Birchfield

★ Must read December 2023

Tracking a mustard bottle while a robot arm picks it up, with the estimated pose rendered at right.

TL;DR One RGBD network estimates and tracks the 6D pose of objects it has never seen, given either a CAD model or about 16 reference images, with no fine-tuning. Trained only on synthetic data, it reaches 83.3 mean AR on three BOP datasets, above the instance-level SurfEmb + ICP at 79.7.

What they do

Pose methods for novel objects usually assume either a CAD model (model-based) or a few reference images (model-free), and either estimate or track. FoundationPose covers all four cases with one RGBD network, trained only on synthetic data and applied to new objects without fine-tuning.

How it works

LLM-aided synthetic data. About 40K Objaverse-LVIS objects plus GSO, retextured by TexFusion from ChatGPT-written appearance descriptions of each object’s category, then rendered in Isaac Sim: about 1.2M images.
Neural object field. Without a CAD model, an SDF and appearance network is fit per object from about 16 reference images in seconds, then turned into a textured mesh. Both setups then feed the same render-and-compare modules.
Refine, then select. After 2D detection, 42 viewpoints × 12 in-plane rotations seed the hypotheses. A transformer refiner compares a rendering at each pose with a crop centered on it and predicts translation and rotation updates. A ranking network scores each hypothesis against the observation, then against all the others, and keeps the best. Tracking runs only the refiner, from the previous frame’s pose.

Results

Model-free, 16 reference images: YCB-Video AUC 91.5 ADD / 97.4 ADD-S vs 42.1 / 88.4 for the fine-tuned FS6D. LINEMOD ADD-0.1d is 99.9 vs 91.5 for FS6D + ICP. With only 4 reference images it still beats FS6D with 16.
Model-based BOP (LM-O, T-LESS, YCB-V): mean AR 83.3 vs 58.6 for MegaPose-RGBD.
Tracking on YCBInEOAT: ADD 93.09 vs 92.66 for the instance-trained se(3)-TrackNet; 93.22 when initialized by its own pose estimate.
On an RTX 3090, estimation takes about 1.3 s per object and tracking runs at about 32 Hz.

Limitations

It relies on an external 2D detector, and false or missing detections often bottleneck it. It fails on textureless, heavily occluded objects with few edge cues. It needs depth, and reference images need known poses (ground truth in the benchmarks, SLAM in the wild).

Videos

AR demo where a tracked handheld box is overlaid with a virtual wooden maze; the inset shows the original video.

Tracking on YCB-Video, with observation, ground truth and FoundationPose side by side.

Figures

From Figure 1: One framework covers 6D pose estimation and tracking of novel objects, in both the model-based (CAD mo... — From Figure 1: One framework covers 6D pose estimation and tracking of novel objects, in both the model-based (CAD model) and model-free (reference images) setups. On each of these four tasks it outperforms prior methods designed for that task.

From Figure 2: Overview. Synthetic training data comes from a 3D model database, an LLM and a diffusion model. In the... — From Figure 2: Overview. Synthetic training data comes from a 3D model database, an LLM and a diffusion model. In the model-free setup an object-centric neural field renders RGBD views for render-and-compare. Pose hypotheses initialized uniformly around the object are refined by the refinement network, then the selection module scores them and the best one is output.

From Figure 4: Pose ranking with and without hierarchical comparison. Comparing all pose hypotheses together gives a ... — From Figure 4: Pose ranking with and without hierarchical comparison. Comparing all pose hypotheses together gives a ranking that better aligns both shape and texture. The true best pose is circled in red.

From Figure 5: Model-free pose estimation on LINEMOD compared with OnePose++ and LatentFusion, on a textureless, self... — From Figure 5: Model-free pose estimation on LINEMOD compared with OnePose++ and LatentFusion, on a textureless, self-occluded glue bottle.

Abstract

We present FoundationPose, a unified foundation model for 6D object pose estimation and tracking, supporting both model-based and model-free setups. Our approach can be instantly applied at test-time to a novel object without fine-tuning, as long as its CAD model is given, or a small number of reference images are captured. We bridge the gap between these two setups with a neural implicit representation that allows for effective novel view synthesis, keeping the downstream pose estimation modules invariant under the same unified framework. Strong generalizability is achieved via large-scale synthetic training, aided by a large language model (LLM), a novel transformer-based architecture, and contrastive learning formulation. Extensive evaluation on multiple public datasets involving challenging scenarios and objects indicate our unified approach outperforms existing methods specialized for each task by a large margin. In addition, it even achieves comparable results to instance-level methods despite the reduced assumptions. Project page: https://nvlabs.github.io/FoundationPose/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong