NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng

★ Must read March 2020

Novel views of a real scene, a mosaic drinking fountain, rendered from a NeRF.

TL;DR An MLP maps a 3D point and viewing direction to density and color, and volume rendering along camera rays turns it into images, so one network per scene is fit from posed photos alone. On realistic synthetic objects NeRF reaches 31.01 PSNR, against 26.05 for the best prior method.

What they do

NeRF represents a static scene as a continuous 5D function: for any 3D location and viewing direction, an MLP returns a volume density and an emitted RGB color. A pixel is rendered by sampling points along its camera ray, querying the network and alpha-compositing the results with classical volume rendering. Because this is differentiable, one network per scene is optimized from RGB images with known poses (from COLMAP for real captures) using only a squared photometric loss.

How it works

Network. Eight 256-wide ReLU layers take the position and output density plus a feature vector; one more layer adds the viewing direction and outputs color. Density depends only on position, which keeps the scene consistent across views, while color can model specularities.
Positional encoding. Each input coordinate is lifted to 2L high-frequency features (L = 10 for position, 4 for direction), which lets the MLP fit fine geometry and texture.
Hierarchical sampling. A coarse network at 64 stratified samples per ray yields a distribution along the ray; 128 more samples drawn from it go to a fine network, focusing queries on visible content.

Results

PSNR on DeepVoxels’ diffuse objects: 40.15 vs 34.38 for LLFF. On their new realistic synthetic set: 31.01 vs 26.05 for Neural Volumes. On 8 real forward-facing phone captures: 26.50 vs 24.13 for LLFF, though LLFF has better LPIPS there (0.212 vs 0.250).
Ablations (synthetic, PSNR): no positional encoding 28.77, no view dependence 27.66, no hierarchical sampling 30.06. With 25 input images NeRF still beats every baseline given 100.
The weights take 5 MB, against over 15 GB of voxel grids for LLFF on one synthetic scene.

Limitations

Each scene needs its own optimization, 100-300k iterations or about 1-2 days on a V100, and rendering takes about 30 s per frame. Scenes must be static with known camera poses. The authors also note that a scene stored in network weights is hard to analyze for expected quality and failure modes.

Videos

Novel views of a real scene, a garden bench with a flower on its armrest.

Figures

From Figure 1: The 100 input views of the synthetic Drums scene, captured on a surrounding hemisphere, and two novel ... — From Figure 1: The 100 input views of the synthetic Drums scene, captured on a surrounding hemisphere, and two novel views rendered from the optimized NeRF.

From Figure 2: 5D coordinates (location and viewing direction) are sampled along camera rays (a) and fed to an MLP th... — From Figure 2: 5D coordinates (location and viewing direction) are sampled along camera rays (a) and fed to an MLP that outputs color and density (b); volume rendering composites them into pixels (c), and the network is optimized against the ground-truth images (d).

From Figure 3: View-dependent color in the Ship scene. Two fixed 3D points, one on the ship's side (orange) and one o... — From Figure 3: View-dependent color in the Ship scene. Two fixed 3D points, one on the ship's side (orange) and one on the water (blue), change appearance between two camera positions (a, b); (c) shows their predicted color over the whole hemisphere of viewing directions.

Abstract

We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(θ, φ)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong