← Paper Digest

SAM-VTP: Video Memory and Measured Geometry for Continuous Traversability Segmentation

Lucas Hoefs, Hui Xie, Senjian An, Fangyi Zhang

November 2026

TL;DR SAM-VTP fine-tunes SAM2's video memory for traversability segmentation, adds slope, roughness and height measured from depth, and trains on continuous rather than binary targets. On a physical robot, the planner found no feasible path 3.3x less often than with a frame-wise SAM2 baseline on a paved course and 6.9x less often on rough ground.

What they do

Vision-based traversability models usually predict each frame on its own, so the drivable region can flicker from frame to frame and leave the planner with no path. SAM-VTP fine-tunes SAM2’s video memory for prompt-free traversability segmentation, feeds in terrain geometry measured from depth, and trains on continuous per-pixel targets between 0 and 1 instead of binary labels.

How it works

  • Video memory. Each frame attends over the features and masks of the 7 most recent frames. The decoder emits a single mask token, and a residual head refines it at full resolution so thin structures such as kerbs survive.
  • Measured geometry. Slope, roughness, height, log range and validity are computed from depth, encoded and added to the coarsest backbone features.
  • Continuous labels, built offline. A hand-set table maps each semantic class to a prior (engineered surfaces 1.0 down to blocked 0.0). Slope and roughness then scale it down, and pixels more than 0.55 m above the local terrain are set to 0. Neither robot driving nor per-pixel annotation is needed.
  • Training. On TartanGround, a synthetic dataset of ground-robot RGB-D sequences, with a fuzzy Jaccard loss plus BCE, focal and temporal-consistency terms.

Results

  • On held-out TartanGround trajectories, fuzzy Jaccard rises from 0.908 to 0.924 and binary IoU from 0.854 to 0.886 over the frame-wise SAM-TP baseline (seed spread below 0.001). Flicker drops by 31%.
  • In an ablation at a shorter training budget, geometry alone adds 0.025 fuzzy Jaccard and memory alone 0.019. Training on binary targets instead costs 0.181.
  • On a JetAcker robot with a RealSense D455f, the planner found no feasible path on 9.9% of cycles against 32.4% for the baseline on a paved course, and 3.0% against 20.6% on rough ground.
  • It runs at 21.4 FPS on the robot’s Jetson Orin Nano.

Limitations

All segmentation accuracy is measured in simulation against the authors’ own targets; the field trials measure planner behaviour, not mask quality. Building targets needs per-pixel semantic labels, which real footage does not have, and the model needs depth (withholding it costs 0.061 fuzzy Jaccard). In the field the baseline saw RGB only, so those trials do not separate memory from depth.

Figures

From Figure 1: The frame-wise SAM-TP baseline (top) against SAM-VTP (bottom) on three consecutive TartanGround frames...
From Figure 1: The frame-wise SAM-TP baseline (top) against SAM-VTP (bottom) on three consecutive TartanGround frames. SAM-TP loses roughly half of the traversable surface at t2 and recovers it at t3, while SAM-VTP holds one coherent region (fuzzy Jaccard 0.69 against 0.92 on this window).
From Figure 2: SAM-VTP keeps SAM2's Hiera-T encoder and memory pathway. The starred blocks are new or modified: a dep...
From Figure 2: SAM-VTP keeps SAM2's Hiera-T encoder and memory pathway. The starred blocks are new or modified: a depth-derived geometry encoder summed into the image features, a learned prompt replacing SAM2's prompt encoder, a decoder emitting a single mask token, and a residual head refining the output at full resolution.
From Figure 3: Continuous traversability supervision for one TartanGround frame. Left to right: semantic labels, heig...
From Figure 3: Continuous traversability supervision for one TartanGround frame. Left to right: semantic labels, height above ground and surface slope from depth, and the final target between 0 and 1.
From Figure 5: Both models replayed over the same recorded robot frames (SAM-TP on RGB, SAM-VTP on RGB-D). On the pav...
From Figure 5: Both models replayed over the same recorded robot frames (SAM-TP on RGB, SAM-VTP on RGB-D). On the paved course (top two rows) SAM-TP breaks into fragments, marking 2% of the image traversable against 32% and 28% for SAM-VTP. On rough terrain (bottom) SAM-TP under-segments a clear path, 6% against 25%.
Abstract

Traversability prediction is a core component of autonomous navigation, yet most vision-based methods operate frame-wise and produce temporally inconsistent predictions. We present SAM-VTP (Video Traversability Prediction), which fine-tunes SAM2's video memory pathway for dense, prompt-free traversability segmentation, conditioning each frame on a persistent memory of prior observations. Rather than inferring terrain geometry from appearance, we measure it: slope, roughness and height are computed from depth and fused into the image features. We also introduce an offline continuous supervision pipeline that generates dense per-pixel traversability targets in [0,1] from depth-derived geometry and semantic priors, requiring no robot deployment and no per-pixel annotation, capturing conditional feasibility that binary labels cannot represent. On TartanGround, a large-scale synthetic ground-robot dataset, SAM-VTP improves fuzzy Jaccard from 0.908 to 0.924 and binary IoU from 0.854 to 0.886 over SAM-TP, a frame-wise SAM2 baseline, against a seed-to-seed spread below 0.001, and cuts frame-to-frame flicker by 31%. In closed-loop trials at two field sites, driving the same planner and controller, the baseline's dropouts leave the planner without a feasible path 3.3x more often on a paved course and 6.9x on rough ground, while SAM-VTP runs at 21.4 FPS on the Orin Nano the robot carries.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.