Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields using Fisher Information

Wen Jiang, Boshu Lei, Kostas Daniilidis

Seminal November 2023

Candidate camera poses around the Lego scene during view selection.

TL;DR FisherRF scores candidate camera poses by the expected information gain computed from the Fisher Information of a radiance field's parameters, without needing the images at those poses. With 10 views on Blender and 3D Gaussian Splatting it reaches 23.68 dB PSNR, against 22.98 dB for ActiveNeRF-style selection and 22.49 dB for random views.

What they do

FisherRF uses Fisher Information to measure how much a candidate camera pose would tell a radiance field about its parameters, and captures the views with the highest expected information gain (EIG). The same quantity drives batch view selection, path choice in active mapping, and per-pixel uncertainty, on 3D Gaussian Splatting and Plenoxels.

How it works

  • No image needed. For an RGB rendering loss, the Hessian of the log-likelihood depends only on the Jacobian of the renderer, so a view can be scored from its camera pose before it is captured.
  • Tractable Hessian. Models have over 200 million parameters, so the Hessian is approximated by its diagonal plus a prior term (Laplace approximation). A CUDA kernel computes it in 11.3 ms, versus 1.1 s for BayesRays’ PyTorch implementation, and candidate views are evaluated at 70 fps with 3D Gaussian Splatting.
  • Batches and paths. Fisher Information is additive, so a greedy algorithm picks batches of views. For active mapping, built on SplaTAM, frontier-based exploration proposes paths and the one with the highest EIG is followed.
  • Uncertainty. The diagonal Hessians of the parameters along each camera ray give a relative per-pixel uncertainty.

Results

  • Blender and Mip-NeRF360, adding views one or four at a time from 4 up to 20: better PSNR, SSIM and LPIPS than random selection and a re-implemented ActiveNeRF.
  • 10-view Blender task: 23.68 dB PSNR and 0.102 LPIPS on 3D Gaussian Splatting, against 22.98 / 0.111 for ActiveNeRF and 22.49 / 0.112 for random. On Plenoxels, 20.67 dB versus 19.77 and 19.95. Selecting by BayesRays uncertainty on Nerfacto did worse than random (14.95 vs 16.43 dB).
  • Active mapping on Matterport3D and Gibson in Habitat: higher completeness than Active-INR, and better rendering than UPEN and Active-INR trajectories when all are trained with 3D Gaussian Splatting.
  • On the Light Field dataset, the uncertainty correlates better with depth error (lower AUSE) than CF-NeRF’s.

Limitations

Static scenes in a confined space only; dynamic radiance fields are left open. The Hessian is reduced to its diagonal, and pixel uncertainty is relative, needing rescaling by depth to be metric. The ActiveNeRF baseline is the authors’ re-implementation.

Figures

From Figure 1: Information gain of a radiance field trained on few views, shown as a heat map on the viewing sphere w...
From Figure 1: Information gain of a radiance field trained on few views, shown as a heat map on the viewing sphere with four candidate views; pixel-wise uncertainty maps (bottom left); and the method used to actively explore and reconstruct an indoor scene (right).
From Figure 2: Active mapping pipeline. RGB-D captures are reconstructed as 3D Gaussians, goal points are picked on t...
From Figure 2: Active mapping pipeline. RGB-D captures are reconstructed as 3D Gaussians, goal points are picked on the map frontier with a shortest path to each, the EIG of each path is computed, and the path with the highest EIG is followed.
From Figure 5: Top-down view of a room-scale scene reconstructed by FisherRF active mapping.
From Figure 5: Top-down view of a room-scale scene reconstructed by FisherRF active mapping.
Abstract

This study addresses the challenging problem of active view selection and uncertainty quantification within the domain of Radiance Fields. Neural Radiance Fields (NeRF) have greatly advanced image rendering and reconstruction, but the cost of acquiring images poses the need to select the most informative viewpoints efficiently. Existing approaches depend on modifying the model architecture or hypothetical perturbation field to indirectly approximate the model uncertainty. However, selecting views from indirect approximation does not guarantee optimal information gain for the model. By leveraging Fisher Information, we directly quantify observed information on the parameters of Radiance Fields and select candidate views by maximizing the Expected Information Gain(EIG). Our method achieves state-of-the-art results on multiple tasks, including view selection, active mapping, and uncertainty quantification, demonstrating its potential to advance the field of Radiance Fields.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.