iNeRF: Inverting Neural Radiance Fields for Pose Estimation

Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, Tsung-Yi Lin

Seminal December 2020

Tracking a real toy car, input video beside the predicted pose and rendering.

TL;DR iNeRF recovers a camera's 6DoF pose from one RGB image by gradient descent on the pixel error between the image and renderings from a trained NeRF, with no mesh model. On unseen ShapeNet cars, using a NeRF predicted from a single view, it reaches 4.39° mean rotation error against 9.27° for a SuperGlue baseline.

What they do

Render-and-compare pose estimation normally needs a watertight mesh of the object; iNeRF uses a NeRF instead. Given an RGB image, known intrinsics, an initial pose guess and a trained NeRF, it freezes the NeRF and runs gradient descent on the camera pose to minimize the photometric error between rendered and observed pixels. The same loop poses unlabeled images for NeRF training and, with pixelNeRF, estimates the pose of objects unseen in training.

How it works

SE(3) optimization. The pose is parameterized in exponential coordinates as a correction applied to the initial estimate, so it stays on SE(3). Putting the center of rotation at the initial estimate reduces coupling between rotation and translation. Adam with an exponentially decaying learning rate.
Ray sampling. Backpropagating through every pixel does not fit in GPU memory, so each step renders a batch of rays. Random rays mostly hit textureless background; sampling only detected interest points converges faster but gets stuck in local minima. Interest region sampling draws rays from dilated regions around the interest points.
Self-supervised NeRF. Train NeRF on posed images, pose the remaining images with iNeRF, then retrain NeRF on all of them.
Category level. pixelNeRF predicts a NeRF from one image of an unseen object; iNeRF aligns it to a second image.

Results

2048 rays per step fit in one forward/backward pass, 150x faster per step than full-image rendering on a 640x480 image. Larger ray batches and interest-region sampling both raise the fraction of poses within 5° or 5 cm.
At batch size 1024, the fraction under 5° rotation error falls from 71% on synthetic scenes to 55% on real LLFF scenes, and under 5 cm from 73% to 39%.
Fern with 25% of poses labeled: PSNR 21.85, rising to 23.89 when iNeRF poses the rest (100% labeled: 24.94).
ShapeNet cars vs. SuperGlue: mean rotation error 4.39° vs. 9.27°, outliers 8.7% vs. 33.3%. On 10 real toy cars, with pixelNeRF trained only on ShapeNet, tracking needs under 10 iterations per frame (about 1 Hz).

Limitations

Lighting changes and occlusion are not modeled and can severely hurt accuracy. 100 optimization steps take about 20 s, too slow for real-time use. It needs an initial pose estimate, and self-supervision needs enough labels: with only 10% of Fern poses, adding iNeRF-posed images lowers PSNR from 18.5 to 15.64.

Videos

Pose tracking on several unseen real toy cars, each from a NeRF predicted from one view.

Fern scene, NeRF rendering overlaid on the photo as the pose estimate converges.

Figures

From Figure 1: iNeRF estimates pose without a mesh model by inverting a neural radiance field. The middle panel shows... — From Figure 1: iNeRF estimates pose without a mesh model by inverting a neural radiance field. The middle panel shows the trajectory of estimated poses (gray) moving toward the ground-truth pose (green); the right panels overlay the rendering at each step on the observed image.

From Figure 2: Pipeline. From the current pose estimate, a set of rays is chosen and rendered through the trained NeR... — From Figure 2: Pipeline. From the current pose estimate, a set of rays is chosen and rendered through the trained NeRF. Because the whole pipeline is differentiable, the residual between rendered and observed pixels is backpropagated to update the pose.

From Figure 3: Three ray-sampling strategies, drawn on the average of the observed image and the current rendering. R... — From Figure 3: Three ray-sampling strategies, drawn on the average of the observed image and the current rendering. Random sampling lands mostly on background (x), which gives no pose gradient. Interest-point sampling picks pixels that are often already aligned (+). Interest-region sampling picks many pixels covered by only one of the two images (o), giving faster and more accurate convergence.

From Figure 5: Fern scene rendered by NeRFs trained on 100%, 50% and 25% of the posed training images, with and witho... — From Figure 5: Fern scene rendered by NeRFs trained on 100%, 50% and 25% of the posed training images, with and without the remaining images added using poses estimated by iNeRF.

Abstract

We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural Radiance Field (NeRF). NeRFs have been shown to be remarkably effective for the task of view synthesis - synthesizing photorealistic novel views of real-world scenes or objects. In this work, we investigate whether we can apply analysis-by-synthesis via NeRF for mesh-free, RGB-only 6DoF pose estimation - given an image, find the translation and rotation of a camera relative to a 3D object or scene. Our method assumes that no object mesh models are available during either training or test time. Starting from an initial pose estimate, we use gradient descent to minimize the residual between pixels rendered from a NeRF and pixels in an observed image. In our experiments, we first study 1) how to sample rays during pose refinement for iNeRF to collect informative gradients and 2) how different batch sizes of rays affect iNeRF on a synthetic dataset. We then show that for complex real-world scenes from the LLFF dataset, iNeRF can improve NeRF by estimating the camera poses of novel images and using these images as additional training data for NeRF. Finally, we show iNeRF can perform category-level object pose estimation, including object instances not seen during training, with RGB images by inverting a NeRF model inferred from a single view.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong