3D Gaussian Splatting for Real-Time Radiance Field Rendering

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis

★ Must read August 2023

Novel-view render of the Garden scene from the optimized Gaussians.

TL;DR Represent a scene as millions of anisotropic 3D Gaussians, start them from SfM points, and render them with a tile-based GPU rasterizer. On the Mip-NeRF360 scenes it nearly matches Mip-NeRF360 quality (PSNR 27.21 vs 27.69) after about 41 minutes of training instead of 48 hours, and renders at 134 fps instead of 0.06 fps.

What they do

Novel-view synthesis from photos of a static scene. The best NeRF, Mip-NeRF360, needs 48 h of training and 10 s per frame; faster methods lose quality. 3D Gaussian Splatting replaces the continuous field with explicit 3D Gaussians, initialized from SfM points and rasterized on the GPU, giving real-time rendering at near Mip-NeRF360 quality.

How it works

Representation. Each Gaussian has a position, opacity, spherical-harmonic color and an anisotropic covariance. The covariance is stored as a scale vector and a rotation quaternion, so gradient descent keeps it valid. The loss is L1 plus D-SSIM.
Adaptive density control. Every 100 iterations, Gaussians with large view-space position gradients are cloned (if small) or split in two (if large). Near-transparent ones are pruned, and opacities are reset every 3000 iterations to suppress floaters. Scenes end with 1-5 million Gaussians.
Tile-based rasterizer. One GPU radix sort orders all Gaussians by 16x16 screen tile and depth, and each tile alpha-blends front to back. The backward pass re-walks the sorted lists, so every Gaussian in a pixel gets a gradient.

Results

Mip-NeRF360 dataset: PSNR 27.21 at 134 fps after 41.5 min of training, vs 27.69 at 0.06 fps after 48 h for Mip-NeRF360 and 25.59 at 9.4 fps for InstantNGP-Big. On Tanks&Temples: 23.14 vs 22.22 for Mip-NeRF360.
After 7K iterations (about 6.5 min) it matches InstantNGP-Big on Mip-NeRF360 (PSNR 25.60 vs 25.59) at 160 fps.
Synthetic Blender scenes: average PSNR 33.32 from 100K random initial points, rendered at 180-300 fps.
Ablations (average PSNR): full 26.05, isotropic Gaussians 25.23, no splitting 23.90, random initialization 20.42, gradients limited to the 10 front-most Gaussians 19.19.

Limitations

Poorly observed regions show artifacts, elongated or “splotchy” Gaussians, and popping when large Gaussians appear; there is no regularization. Memory is much higher than for NeRFs: 270-734 MB per model on average, and over 20 GB peak GPU memory when training large scenes.

Videos

Novel-view render of the indoor Playroom scene.

Figures

From Figure 1: Bicycle test view. With training times comparable to InstantNGP the method reaches similar quality; tr... — From Figure 1: Bicycle test view. With training times comparable to InstantNGP the method reaches similar quality; trained for 51 min it is slightly better than Mip-NeRF360, while rendering in real time.

From Figure 2: Pipeline. Sparse SfM points initialize a set of 3D Gaussians, which are optimized and adaptively densi... — From Figure 2: Pipeline. Sparse SfM points initialize a set of 3D Gaussians, which are optimized and adaptively densified through a differentiable tile-based rasterizer.

From Figure 4: Adaptive densification. Where geometry is under-covered, a small Gaussian is cloned; where one large G... — From Figure 4: Adaptive densification. Where geometry is under-covered, a small Gaussian is cloned; where one large Gaussian covers fine geometry, it is split in two.

From Figure 3: A rendered view (left) and the optimized Gaussians shrunk by 60% (far right), showing the anisotropic ... — From Figure 3: A rendered view (left) and the optimized Gaussians shrunk by 60% (far right), showing the anisotropic shapes that represent fine geometry.

Abstract

Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong