Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Maps

Timothy Chen, Ola Shorinwa, Joseph Bruno, Aiden Swann, Javier Yu, Weijia Zeng, Keiko Nagami, Philip Dames, Mac Schwager

Seminal March 2024

Overhead view of the real drone flying open-loop through the Maze to the keyboard.

TL;DR Plans and localizes a drone directly in a Gaussian Splatting map. Polytope corridors built from the splat's ellipsoids give provably safe Bézier trajectories, and RGB-only PnP against splat renders gives the pose. Re-planning runs at over 2 Hz and localization at about 25 Hz, and only one of 120 hardware flights collided.

What they do

Splat-Nav navigates a drone inside a pre-trained Gaussian Splatting (GSplat) map using only an RGB camera. Each Gaussian is treated as an ellipsoid, which gives the planner exact collision geometry to plan against. The same splat is also rendered to localize the drone. Goals can be language queries such as “microwave”, answered by a semantic GSplat.

How it works

Collision test. A GPU-batched ellipsoid-ellipsoid test, solved by bisection, never misses a collision. It also extends to an ellipsoid sweeping along a line segment.
Splat-Plan. The splat is over-approximated as an occupancy grid, and Dijkstra finds a seed path. Supporting hyperplanes from nearby ellipsoids build a corridor of safe polytopes around it. A QP then fits Bézier curves whose control points stay inside the corridor, so the whole trajectory is safe.
Splat-Loc. It renders RGB and depth at the prior pose and matches SuperPoint/LightGlue features against the camera image. PnP with RANSAC then gives the pose. For global initialization, it registers splat means against a point cloud with color-augmented FPFH features.

Results

Planning in simulation. Four scenes, each with a dense and a sparse splat, and 100 start-goal pairs per setting. Splat-Plan never fails and stays collision-free with respect to the GSplat geometry. Four point-cloud corridor (SFC) variants sometimes collide or fail, and Splat-Plan’s paths are as short or shorter.
Localization in simulation. With a 20° / 0.1 m initial error, Splat-Loc-Glue is under 0.06° and 4 mm off in under 42 ms per frame. Photometric GS-Loc takes 36.15 s per frame.
Hardware. 120 flights covered four language goals, three control schemes and 10 repeats. Every flight was collision-free except one Splat-Loc flight whose camera faced a poorly reconstructed edge of the scene. Splat-Loc was about 3° and 4 cm off motion capture, comparable to VIO, and needed no frame alignment. VIO had to be re-zeroed every run. Fast flights reached 1.5 m/s.

Limitations

Maps are pre-built and static. Safety depends on the map: an obstacle missing from the splat cannot be avoided. Splat-Loc needs visual features. Splat-Plan and Splat-Loc need about 10 GB of GPU memory, so they ran on a desktop RTX 4090 linked to the drone over WiFi.

Videos

Closed-loop re-planning flight to the beachball.

Closed-loop re-planning flight to the microwave.

Figures

From Figure 1: The Splat-Nav pipeline. Splat-Plan uses an ellipsoid-ellipsoid collision test to find a safe flight co... — From Figure 1: The Splat-Nav pipeline. Splat-Plan uses an ellipsoid-ellipsoid collision test to find a safe flight corridor through the GSplat and plans a spline through it. Splat-Loc localizes the robot from RGB images alone, using PnP against a point cloud rendered from the GSplat. A language-embedded GSplat turns goals like "go to the microwave" into positions.

From Figure 3: Splat-Plan. The GSplat's ellipsoidal collision geometry is converted into a binary occupancy grid, and... — From Figure 3: Splat-Plan. The GSplat's ellipsoidal collision geometry is converted into a binary occupancy grid, and graph search finds a seed path through it. Connected polytopes form a corridor around that path. A quadratic program then places a sequence of Bézier curves entirely inside the corridor.

From Figure 9: Ground-truth drone trajectories projected onto the Maze GSplat. Columns show four language goals flown... — From Figure 9: Ground-truth drone trajectories projected onto the Maze GSplat. Columns show four language goals flown open-loop, with closed-loop VIO, with Splat-Loc, and at high speed. Crosses mark the one failed Splat-Loc flight.

Abstract

We present Splat-Nav, a real-time robot navigation pipeline for Gaussian Splatting (GSplat) scenes, a powerful new 3D scene representation. Splat-Nav consists of two components: 1) Splat-Plan, a safe planning module, and 2) Splat-Loc, a robust vision-based pose estimation module. Splat-Plan builds a safe-by-construction polytope corridor through the map based on mathematically rigorous collision constraints and then constructs a Bézier curve trajectory through this corridor. Splat-Loc provides real-time recursive state estimates given only an RGB feed from an on-board camera, leveraging the point-cloud representation inherent in GSplat scenes. Working together, these modules give robots the ability to recursively re-plan smooth and safe trajectories to goal locations. Goals can be specified with position coordinates, or with language commands by using a semantic GSplat. We demonstrate improved safety compared to point cloud-based methods in extensive simulation experiments. In a total of 126 hardware flights, we demonstrate equivalent safety and speed compared to motion capture and visual odometry, but without a manual frame alignment required by those methods. We show online re-planning at more than 2 Hz and pose estimation at about 25 Hz, an order of magnitude faster than Neural Radiance Field (NeRF)-based navigation methods, thereby enabling real-time navigation. We provide experiment videos on our project page at https://chengine.github.io/splatnav/. Our codebase and ROS nodes can be found at https://github.com/chengine/splatnav.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong