ActiveNeRF: Learning where to See with Uncertainty Estimation

Xuran Pan, Zihang Lai, Shiji Song, Gao Huang

Seminal September 2022

TL;DR ActiveNeRF makes NeRF predict a variance for each point's color, then adds the new views that would shrink that uncertainty the most to a small training set. With 4 starting views plus 16 chosen ones on synthetic scenes, it reaches 26.24 PSNR versus 24.25 for random views and 26.00 for furthest-view selection.

What they do

NeRF trained on few images, or on images that miss part of the scene, generalizes poorly and can collapse to zero density in the unseen regions. ActiveNeRF works under a fixed capture budget: it adds an uncertainty estimate to NeRF and uses it to decide which new views are worth capturing.

How it works

Uncertain radiance. An extra MLP branch with a softplus predicts a variance for each point’s color. Because points are independent and volume rendering is a weighted sum, each ray’s color is also Gaussian. Training minimizes its negative log-likelihood plus a density-sparsity term that prevents blurry surfaces. Only the fine network gets this branch.
Acquisition. For a candidate view, Bayesian updating gives the posterior variance of every sampled point without knowing the new image’s colors. The summed drop from prior to posterior variance scores the view, and the top-k candidates are added.
Using new views. Either keep training on them (continuous learning) or plug their colors into the posterior mean without training (Bayesian estimation). Candidates can be scored at reduced resolution to save time.

Results

Fixed training sets: with all images it is on par with NeRF (30.45 vs 31.01 PSNR on synthetic scenes, LPIPS 0.072 vs 0.081). With 5 images it reaches 23.23 vs 21.14 PSNR.
Active setting with 20 total views: continuous learning gets 26.24 PSNR on synthetic scenes vs 24.25 random and 26.00 furthest-view sampling, and 23.12 vs 20.65 and 22.41 on LLFF. Training takes 2.2 h vs 2.0 h.
Bayesian estimation takes 30 min but drops to 25.67 PSNR, below furthest-view sampling.
Scoring candidates at 1/5 resolution keeps 26.12 PSNR; at 1/20 it falls to 24.67.

Limitations

The paper has no limitations section. Its active-learning experiments choose among held-out dataset images rather than capturing new ones. With all training images, the uncertainty model scores slightly below NeRF in PSNR on both datasets. With 10 total views, the training-free Bayesian variant falls below random selection on synthetic scenes.

Figures

From Figure 1: ActiveNeRF estimates scene uncertainty, picks the novel view that would reduce it the most, adds that ... — From Figure 1: ActiveNeRF estimates scene uncertainty, picks the novel view that would reduce it the most, adds that view to the training set and repeats, which improves synthesis and lowers uncertainty.

From Figure 4: The pipeline. ActiveNeRF is trained on the initial views, candidate views are scored by how much they ... — From Figure 4: The pipeline. ActiveNeRF is trained on the initial views, candidate views are scored by how much they would reduce uncertainty, and the best one is captured and added to the training set.

From Figure 5: NeRF and ActiveNeRF renders with uncertainty maps on a synthetic and a real scene, trained on all imag... — From Figure 5: NeRF and ActiveNeRF renders with uncertainty maps on a synthetic and a real scene, trained on all images or on 10. ActiveNeRF is clearly better in the low-shot rows, matches NeRF with all images, and its uncertainty drops as more images are used.

Abstract

Recently, Neural Radiance Fields (NeRF) has shown promising performances on reconstructing 3D scenes and synthesizing novel views from a sparse set of 2D images. Albeit effective, the performance of NeRF is highly influenced by the quality of training samples. With limited posed images from the scene, NeRF fails to generalize well to novel views and may collapse to trivial solutions in unobserved regions. This makes NeRF impractical under resource-constrained scenarios. In this paper, we present a novel learning framework, ActiveNeRF, aiming to model a 3D scene with a constrained input budget. Specifically, we first incorporate uncertainty estimation into a NeRF model, which ensures robustness under few observations and provides an interpretation of how NeRF understands the scene. On this basis, we propose to supplement the existing training set with newly captured samples based on an active learning scheme. By evaluating the reduction of uncertainty given new inputs, we select the samples that bring the most information gain. In this way, the quality of novel view synthesis can be improved with minimal additional resources. Extensive experiments validate the performance of our model on both realistic and synthetic scenes, especially with scarcer training data. Code will be released at https://github.com/LeapLabTHU/ActiveNeRF.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong