Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM

Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, Jonathon Luiten

Seminal December 2023

Online reconstruction of ScanNet++ scene S1 as the camera is tracked.

TL;DR SplaTAM is a dense RGB-D SLAM system that uses 3D Gaussian Splatting as its map, tracking the camera and growing the map by differentiable rendering. It tracks ScanNet++ with 1.2 cm trajectory error where Point-SLAM and ORB-SLAM3 fail, and cuts Replica error from 0.52 cm to 0.36 cm.

What they do

SplaTAM does dense SLAM from a single unposed RGB-D camera, using 3D Gaussians as the map. Because Gaussians rasterize fast, both camera tracking and mapping can use a dense loss over every pixel of the rendered image, where earlier neural SLAM methods sample a few hundred pixels.

How it works

  • Map. Simplified Gaussians: isotropic, with view-independent color, so each has 8 parameters (color, center, radius, opacity). Besides color and depth, the renderer outputs a silhouette, the accumulated opacity per pixel, which marks the parts of the view the map already covers.
  • Tracking. The new pose starts from a constant-velocity prediction, then gradient descent on an L1 depth and color loss with the map frozen. The loss only counts pixels with silhouette above 0.99, i.e. well-mapped regions.
  • Densification. New Gaussians are added where the silhouette is below 0.5, or where the observed depth lies well in front of the rendered depth.
  • Map update. With poses fixed, Gaussians are optimized over the current frame and a few keyframes that overlap it most, with an added SSIM term; near-transparent or oversized Gaussians are removed.

Results

  • Trajectory error (ATE RMSE): 1.2 cm on ScanNet++, where Point-SLAM (343.8 cm) and ORB-SLAM3 (158.2 cm) fail; 0.36 cm on Replica vs 0.52 cm for Point-SLAM; 5.48 cm vs 8.92 cm on TUM-RGBD. On the original ScanNet it is on par with prior dense methods (11.88 cm).
  • In a new ScanNet++ novel-view benchmark, held-out views reach 24.41 dB PSNR and about 2 cm depth error; Point-SLAM gets 11.91 dB.
  • Ablations on Replica Room 0: without the silhouette mask tracking fails (115.8 cm), and without velocity propagation error rises from 0.27 cm to 2.95 cm.
  • It renders roughly 1000x more pixels per iteration than NICE-SLAM and Point-SLAM at similar runtime; SplaTAM-S is 5x faster at 0.39 cm ATE.

Limitations

Sensitive to motion blur, large depth noise and aggressive rotation. Requires known camera intrinsics and dense depth. Scaling to large scenes is left to future work.

Videos

Novel-view renders of the reconstructed ScanNet++ scene S2.
SplaTAM novel-view rendering of Replica Room 0.
Full video (YouTube)

Figures

From Figure 1: Left, the 3D Gaussian map with the train (SLAM-input) and novel-view camera frustums; SplaTAM localize...
From Figure 1: Left, the 3D Gaussian map with the train (SLAM-input) and novel-view camera frustums; SplaTAM localizes to sub-cm accuracy despite large motion between cameras in a texture-less room. Right, train and novel views rendered at 400 FPS at 876x584.
From Figure 2: Each timestep takes the new RGB-D frame and the Gaussian map built so far. (1) Estimate the camera pos...
From Figure 2: Each timestep takes the new RGB-D frame and the Gaussian map built so far. (1) Estimate the camera pose with silhouette-guided differentiable rendering, (2) add new Gaussians based on the rendered silhouette and input depth, (3) update the map with differentiable rendering.
From Figure S.1: Reconstruction of ScanNet++ scene S2 with SplaTAM's estimated poses (green frustums, red trajectory)...
From Figure S.1: Reconstruction of ScanNet++ scene S2 with SplaTAM's estimated poses (green frustums, red trajectory) overlaid on the ground truth (blue frustums and trajectory).
Abstract

Dense simultaneous localization and mapping (SLAM) is crucial for robotics and augmented reality applications. However, current methods are often hampered by the non-volumetric or implicit way they represent a scene. This work introduces SplaTAM, an approach that, for the first time, leverages explicit volumetric representations, i.e., 3D Gaussians, to enable high-fidelity reconstruction from a single unposed RGB-D camera, surpassing the capabilities of existing methods. SplaTAM employs a simple online tracking and mapping system tailored to the underlying Gaussian representation. It utilizes a silhouette mask to elegantly capture the presence of scene density. This combination enables several benefits over prior representations, including fast rendering and dense optimization, quickly determining if areas have been previously mapped, and structured map expansion by adding more Gaussians. Extensive experiments show that SplaTAM achieves up to 2x superior performance in camera pose estimation, map construction, and novel-view synthesis over existing methods, paving the way for more immersive high-fidelity SLAM applications.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.