Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

iMAP: Implicit Mapping and Positioning in Real-Time

Edgar Sucar, Shikun Liu, Joseph Ortiz, Andrew J. Davison

Seminal March 2021

TL;DR A real-time RGB-D SLAM system whose only map is a small MLP trained live from scratch, tracking at 10 Hz and mapping at 2 Hz. On Replica it fills in more of the room than TSDF fusion (79% vs 75% completion ratio) with about 13 keyframes and 60x less memory.

What they do

iMAP is a dense SLAM system for a handheld RGB-D camera. Its only scene representation is an MLP that starts from random weights and is trained live, with no prior data. A tracking process aligns each frame against depth and colour rendered from the network. In parallel, a mapping process keeps training the network on a small set of keyframes that acts as a replay buffer against forgetting. The whole system is written in PyTorch and runs on a single desktop CPU/GPU.

How it works

  • Scene network. A NeRF-style MLP (4 hidden layers of 256 units) with a learnable Gaussian Fourier positional embedding maps a 3D point to colour and volume density, without viewing direction. Volume rendering along rays gives depth, colour and depth variance.
  • Losses. An L1 photometric loss plus a depth loss normalised by the rendered depth variance, minimised with Adam. Tracking optimises only the live pose against the frozen network. Mapping jointly optimises the network and the keyframe poses.
  • Keyframes. A frame becomes a keyframe when less than 65% of its pixels are explained by a snapshot of the map. Each mapping step uses 5 frames: 3 keyframes sampled by loss, the last keyframe and the live frame.
  • Active sampling. Only 200 pixels per image are rendered. Each image is split into an 8x8 grid, and samples are moved towards high-loss cells and high-loss keyframes.

Results

  • Tracking runs at 10 Hz and mapping at 2 Hz.
  • Replica (8 scenes): average completion ratio of 79.1%, against 75.1% for TSDF fusion given the same poses (11 points higher in office-3). This uses 13 keyframes on average. Accuracy is worse: 4.43 cm against 3.45 cm.
  • The map takes 1.04 MB, against 67.10 MB for a 256³ TSDF volume.
  • TUM RGB-D tracking error is 2.0-5.8 cm, behind BAD-SLAM (1.1-1.73 cm) and ORB-SLAM2 (0.4-1.6 cm). On real data iMAP fills holes where the depth sensor fails, such as black, reflective and transparent surfaces.

Limitations

Tracking is less accurate than BAD-SLAM and ORB-SLAM2, and the geometry is less accurate than TSDF fusion. A coarse map forms in seconds, but fine detail takes minutes of training. The authors name more structured, compositional scene representations as future work.

Figures

From Figure 1: Room reconstruction from real-time iMAP with an Azure Kinect RGB-D camera, showing the watertight scen...
From Figure 1: Room reconstruction from real-time iMAP with an Azure Kinect RGB-D camera, showing the watertight scene model, camera tracking and the automatic keyframe set.
From Figure 2: iMAP system pipeline. A tracking process estimates the pose of each RGB-D frame; a mapping process add...
From Figure 2: iMAP system pipeline. A tracking process estimates the pose of each RGB-D frame; a mapping process adds new keyframes and jointly optimises them with the implicit network.
From Figure 4: Keyframe active sampling. A loss distribution over the registered keyframes picks a bounded window of ...
From Figure 4: Keyframe active sampling. A loss distribution over the registered keyframes picks a bounded window of keyframes (red boxes) and allocates pixel samples within each.
From Figure 11: Hole filling of iMAP (top) against BAD-SLAM (bottom).
From Figure 11: Hole filling of iMAP (top) against BAD-SLAM (bottom).
Abstract

We show for the first time that a multilayer perceptron (MLP) can serve as the only scene representation in a real-time SLAM system for a handheld RGB-D camera. Our network is trained in live operation without prior data, building a dense, scene-specific implicit 3D model of occupancy and colour which is also immediately used for tracking. Achieving real-time SLAM via continual training of a neural network against a live image stream requires significant innovation. Our iMAP algorithm uses a keyframe structure and multi-processing computation flow, with dynamic information-guided pixel sampling for speed, with tracking at 10 Hz and global map updating at 2 Hz. The advantages of an implicit MLP over standard dense SLAM techniques include efficient geometry representation with automatic detail control and smooth, plausible filling-in of unobserved regions such as the back surfaces of objects.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.