Vision-Only Robot Navigation in a Neural Radiance World

Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, Mac Schwager

Seminal October 2021

Planning a quadrotor trajectory between a start and goal in the Stonehenge NeRF.

TL;DR A robot plans through a pre-trained NeRF by minimizing NeRF density along a differentially flat trajectory and localizes by matching its RGB images to NeRF renders, replanning in a loop. In simulated scenes a quadrotor reaches the goal without collision where open-loop execution crashes, but each 0.1 s step still takes about 2 s to replan and 4 s to estimate state.

What they do

Given a NeRF of a scene trained offline, a robot plans dynamically feasible, collision-free trajectories through it and localizes itself using only an onboard RGB camera. A trajectory optimizer and a NeRF-based state filter run together in a receding-horizon loop. All experiments are in simulation: NeRFs are trained on images rendered from meshes of a playground, a church and Stonehenge, and results are checked against the ground-truth meshes.

How it works

Collision cost from density. NeRF density is taken as a proxy for the chance of stopping a mass particle. Integrating density over a grid of points covering the robot body, along the path each point travels, bounds the collision probability.
Flatness-based planner. Decision variables are dense flat-output waypoints (position and yaw for a quadrotor); full pose and control inputs follow from the dynamics. Adam minimizes collision cost plus control effort, starting from a straight line or coarse A*.
Pose filter. Each step propagates mean and covariance through the dynamics, then minimizes a process loss plus a photometric loss between 256 ORB-guided image pixels and NeRF renders, re-projecting onto SE(3) after every gradient step. The Hessian gives the posterior covariance.
Replanning. The posterior mean is the new start state and the previous waypoints hot-start the next optimization.

Results

NeRF collision loss tracks ground-truth mesh intersection during optimization; by iteration 49 the playground trajectory has zero intersection.
In Stonehenge the planner has lower control effort, NeRF collision cost and failure rate than minimum-snap on A* waypoints and RRT on a marching-cubes mesh. It also turns a couch-shaped omnidirectional robot to fit through a narrow gap in the church.
Over 100 trials with 2 cm and 0.02 rad dynamics noise, the filter beats dynamics-initialized iNeRF on almost every metric. With feedback the quadrotor reaches the goal collision-free; open-loop execution collides.
On an RTX 3090: initial plan 20 s (2500 iterations), each replan 2 s (250 iterations), each state estimate 4 s (300 steps), for a 0.1 s timestep.

Limitations

Simulation only, with no real robot. Far from real time; NeRF rendering speed is the bottleneck. Collision is a cost rather than a constraint, the scene must be static and its NeRF trained beforehand.

Videos

Closed-loop flight in Stonehenge, replanning the remaining trajectory as the drone moves.

Onboard camera view of Stonehenge used by the state estimator.

Figures

From Figure 1: A drone navigating through the interior of a church modeled as a NeRF, using a monocular camera. The t... — From Figure 1: A drone navigating through the interior of a church modeled as a NeRF, using a monocular camera. The trajectory minimizes a NeRF-based collision metric and is replanned as the drone updates its state estimate from captured images.

From Figure 2: The pipeline. At each timestep the planner optimizes a trajectory from the current mean state estimate... — From Figure 2: The pipeline. At each timestep the planner optimizes a trajectory from the current mean state estimate against a NeRF-based collision metric, the robot applies the first control action and receives a camera image, and the state estimator uses the NeRF as a measurement model to update its belief over the new state.

From Figure 7: Example trajectories through Stonehenge from the proposed planner (green), RRT (blue) and minimum-snap... — From Figure 7: Example trajectories through Stonehenge from the proposed planner (green), RRT (blue) and minimum-snap (red). The planner moves its waypoints to get a smooth, collision-free path; minimum-snap follows the A* initialization exactly and clips a column, and RRT avoids collision but its erratic path needs high control effort.

From Figure 9: Rotational, translational, angular-rate and velocity estimation errors, averaged over 100 trials with ... — From Figure 9: Rotational, translational, angular-rate and velocity estimation errors, averaged over 100 trials with one-standard-deviation bands. The proposed filter (red) has lower rotational, translational and velocity error, with lower variance, than iNeRF initialized through the dynamics (green).

Abstract

Neural Radiance Fields (NeRFs) have recently emerged as a powerful paradigm for the representation of natural, complex 3D scenes. NeRFs represent continuous volumetric density and RGB values in a neural network, and generate photo-realistic images from unseen camera viewpoints through ray tracing. We propose an algorithm for navigating a robot through a 3D environment represented as a NeRF using only an on-board RGB camera for localization. We assume the NeRF for the scene has been pre-trained offline, and the robot's objective is to navigate through unoccupied space in the NeRF to reach a goal pose. We introduce a trajectory optimization algorithm that avoids collisions with high-density regions in the NeRF based on a discrete time version of differential flatness that is amenable to constraining the robot's full pose and control inputs. We also introduce an optimization based filtering method to estimate 6DoF pose and velocities for the robot in the NeRF given only an onboard RGB camera. We combine the trajectory planner with the pose filter in an online replanning loop to give a vision-based robot navigation pipeline. We present simulation results with a quadrotor robot navigating through a jungle gym environment, the inside of a church, and Stonehenge using only an RGB camera. We also demonstrate an omnidirectional ground robot navigating through the church, requiring it to reorient to fit through the narrow gap. Videos of this work can be found at https://mikh3x4.github.io/nerf-navigation/ .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong