← Paper Digest

Bridging vision-language reasoning and learned control for legged navigation in the wild

Takahiro Miki, Robin Schmid, Manthan Patel, Turcan Tuna, Julia Richter, Julian Nubert, Joonho Lee, Matthias Müller, Marco Hutter

November 2026

ANYmal D walking autonomously on the Zermatt descent.

TL;DR ROAM has a VLM label the scene with risk, preference and steppability values, segments those labels into maps, and feeds them to RL locomotion and navigation policies on an ANYmal D. The robot hiked 10.1 km in Zermatt with a 96.7% autonomy rate and read construction signs to detour around a closed path.

What they do

Geometric traversability misses hazards like water or tall grass and ignores norms like staying on the trail. ROAM gives a legged robot this common sense without asking a VLM for coordinates or motor commands. The VLM only names what it sees and scores it; segmentation, mapping and RL policies handle the spatial part. The system runs on an ANYmal D in alpine, urban and indoor settings.

How it works

  • Scene evaluation. Every 10 s, GPT-4 Vision gets the front and rear camera images plus a prompt describing the robot. It returns labels with risk, steppable and preferred-area values (0 to 1), plus a memory note fed into later queries. A YOLO detector pauses the robot when it sees a sign so the VLM can read it.
  • Maps. Open-vocabulary segmentation runs at 6 Hz and turns the labels into per-pixel value images. These are fused into a 4 cm, 8 x 8 m elevation map as extra layers.
  • Policies. A teacher-student locomotion policy takes height and steppable scans and is penalized for non-steppable footholds. An RL local navigation policy takes risk and elevation layers, and non-preferred areas get a virtual height increase. Both are trained in simulation on these abstract inputs; the global route comes from hiking-path data.

Results

  • Zermatt: 10.1 km with a 1220 m descent at 96.7% autonomy in time, with two interventions at construction zones. Seealpsee: 2.38 km at 97.5%.
  • Mannequin on the path, 10 trials: ROAM reached the goal every time with no collisions. The geometry-only baseline hit it every time; ViNT reached the goal 60% of the time and collided in 90%.
  • Path generation on 27 hand-labeled images: 99.1% of ROAM trajectory pixels lay in safe regions and 86.8% in preferred ones. Asking the VLM for pixel coordinates gave at most 71.4% and 39.8%.

Limitations

Segmentation sometimes fails to match the VLM’s labels: it missed a fence the VLM had named. The system needs a global reference path, depends on a cloud VLM and network, and was tested mainly in daylight. Field runs were done once per method.

Videos

Placing feet around cans the VLM marked non-steppable, with the steppable map inset.
Full video (YouTube)
Full video (YouTube)

Figures

From Figure 1: The ROAM pipeline. A VLM turns RGB images and a robot description into object labels with risk, prefer...
From Figure 1: The ROAM pipeline. A VLM turns RGB images and a robot description into object labels with risk, preferred-area and steppable values. Open-vocabulary segmentation converts these into semantic images, faster than the VLM runs, which are fused with elevation maps for the RL policies.
From Figure 3: Long-distance hikes in the Swiss Alps. (A) Zermatt, 10.1 km with a 1220 m elevation change over gravel...
From Figure 3: Long-distance hikes in the Swiss Alps. (A) Zermatt, 10.1 km with a 1220 m elevation change over gravel, rocks, grass and stretches near a lake and train tracks, with gradients up to about 30%. (B) Seealpsee, 2.38 km around a lake, with a narrow trail by the water, cows, gravel and forest. Photos (i) to (vi) match the labeled map locations.
From Figure 4: ROAM against a geometry-only planner and ViNT on a 1.2 km course. (A) A global path offset into bushes...
From Figure 4: ROAM against a geometry-only planner and ViNT on a 1.2 km course. (A) A global path offset into bushes and grass. (B) A mannequin lying on the path. (C) Construction signs: the paths of the three methods, and ROAM's risk and preferred-area images before and after reading the "ROAD WORK AHEAD" and "END ROAD WORK" signs.
Abstract

Autonomous navigation in diverse real-world environments remains one of the greatest challenges in robotics. It demands both robust low-level motion control to handle diverse terrain and high-level reasoning to interpret environmental cues. We present ROAM (Robust Open-world Autonomous Mobility), a system that enables legged robots to perform kilometer-scale navigation in complex alpine and urban settings. ROAM combines the world knowledge of a Vision-Language Model (VLM), pre-trained on internet-scale data, with robust locomotion and local planning policies learned in simulation via reinforcement learning (RL). The VLM provides common-sense reasoning and semantic interpretation of the scene, assessing risks, interpreting signs, and producing semantic labels with associated navigation costs. An open-vocabulary segmentation module converts these labels into a dense, real-time risk map, which conditions two pre-trained RL policies responsible for agile locomotion and local navigation, effectively coupling high-level VLM reasoning with low-level physical control. Real-world experiments showed the system's ability to autonomously navigate a wide range of complex real-world environments. The robot completes kilometer-long hikes with minimal intervention, interprets signs to avoid danger, and evaluates its environment to identify safe areas on which to step. This work represents a step toward the reliable deployment of systems in unstructured environments. However, it also underscores the challenges that remain in VLM reliability and spatial reasoning, indicating that foundational models are a crucial component but not yet a complete solution for navigation in the wild.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.