SAM-VTP: Video Memory and Measured Geometry for Continuous Traversability Segmentation

TL;DR SAM-VTP fine-tunes SAM2's video memory for traversability segmentation, adds slope, roughness and height measured from depth, and trains on continuous rather than binary targets. On a physical robot, the planner found no feasible path 3.3x less often than with a frame-wise SAM2 baseline on a paved course and 6.9x less often on rough ground.
What they do
Vision-based traversability models usually predict each frame on its own, so the drivable region can flicker from frame to frame and leave the planner with no path. SAM-VTP fine-tunes SAM2’s video memory for prompt-free traversability segmentation, feeds in terrain geometry measured from depth, and trains on continuous per-pixel targets between 0 and 1 instead of binary labels.
How it works
- Video memory. Each frame attends over the features and masks of the 7 most recent frames. The decoder emits a single mask token, and a residual head refines it at full resolution so thin structures such as kerbs survive.
- Measured geometry. Slope, roughness, height, log range and validity are computed from depth, encoded and added to the coarsest backbone features.
- Continuous labels, built offline. A hand-set table maps each semantic class to a prior (engineered surfaces 1.0 down to blocked 0.0). Slope and roughness then scale it down, and pixels more than 0.55 m above the local terrain are set to 0. Neither robot driving nor per-pixel annotation is needed.
- Training. On TartanGround, a synthetic dataset of ground-robot RGB-D sequences, with a fuzzy Jaccard loss plus BCE, focal and temporal-consistency terms.
Results
- On held-out TartanGround trajectories, fuzzy Jaccard rises from 0.908 to 0.924 and binary IoU from 0.854 to 0.886 over the frame-wise SAM-TP baseline (seed spread below 0.001). Flicker drops by 31%.
- In an ablation at a shorter training budget, geometry alone adds 0.025 fuzzy Jaccard and memory alone 0.019. Training on binary targets instead costs 0.181.
- On a JetAcker robot with a RealSense D455f, the planner found no feasible path on 9.9% of cycles against 32.4% for the baseline on a paved course, and 3.0% against 20.6% on rough ground.
- It runs at 21.4 FPS on the robot’s Jetson Orin Nano.
Limitations
All segmentation accuracy is measured in simulation against the authors’ own targets; the field trials measure planner behaviour, not mask quality. Building targets needs per-pixel semantic labels, which real footage does not have, and the model needs depth (withholding it costs 0.061 fuzzy Jaccard). In the field the baseline saw RGB only, so those trials do not separate memory from depth.
Figures




Abstract
Traversability prediction is a core component of autonomous navigation, yet most vision-based methods operate frame-wise and produce temporally inconsistent predictions. We present SAM-VTP (Video Traversability Prediction), which fine-tunes SAM2's video memory pathway for dense, prompt-free traversability segmentation, conditioning each frame on a persistent memory of prior observations. Rather than inferring terrain geometry from appearance, we measure it: slope, roughness and height are computed from depth and fused into the image features. We also introduce an offline continuous supervision pipeline that generates dense per-pixel traversability targets in [0,1] from depth-derived geometry and semantic priors, requiring no robot deployment and no per-pixel annotation, capturing conditional feasibility that binary labels cannot represent. On TartanGround, a large-scale synthetic ground-robot dataset, SAM-VTP improves fuzzy Jaccard from 0.908 to 0.924 and binary IoU from 0.854 to 0.886 over SAM-TP, a frame-wise SAM2 baseline, against a seed-to-seed spread below 0.001, and cuts frame-to-frame flicker by 31%. In closed-loop trials at two field sites, driving the same planner and controller, the baseline's dropouts leave the planner without a feasible path 3.3x more often on a paved course and 6.9x on rough ground, while SAM-VTP runs at 21.4 FPS on the Orin Nano the robot carries.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
