MOSU2: Long-range Navigation System with Multi-modal Scene Understanding

Jing Liang, Zhenyu Hou, Senthurbavan Kirubaharan, Md Safwan Mondal, Ashok Gudivada, Xuesu Xiao, Dinesh Manocha

November 2026

Real-world run: the robot crosses a signalized street on the pedestrian signal.

TL;DR An outdoor navigation system where a VLM first screens each scene from one image and, only when needed, pulls positions and velocities of nearby road users from a local map before picking a trajectory. Across real-world social and traffic scenarios, mean navigation success rises from 61.8% to 94.2% over its predecessor MOSU, at about 2.2 times the decision latency.

What they do

Robots that travel long distances outdoors must stay on terrain they can handle and also decide when to proceed, yield or detour around pedestrians, cyclists, cars, traffic signals and construction. A VLM looking at one image cannot tell whether a car is slowing down. MOSU2 extends the MOSU navigation system so the VLM can ask a local map for the measured geometry and motion of the things it sees, and does so only when the scene calls for it.

How it works

Inherited from MOSU. A public routing service gives GPS subgoals about 35 m apart. A CVAE generates candidate trajectories from LiDAR and velocity history, and SAM3 open-vocabulary segmentation scores how much of each trajectory lies on terrain the robot type can traverse. Low-scoring candidates are filtered out.
VLM-retrievable local map. Segmented objects (cars, pedestrians, bicycles, traffic lights) are given metric positions with LiDAR and stored in a grid of 20 m radius and 0.2 m resolution, with velocities estimated across frames. The map is queried by semantic label and region.
Two-stage selection. Stage one sees a single image with numbered trajectories, picks the default one toward the subgoal and decides whether more evidence is needed. If so, stage two reads 5 recent frames plus the requested map data and outputs a trajectory and a velocity mode (normal, slow, stop).

Results

Real-robot trials in five scenario groups, with Husky, Unitree Go2 and Diablo platforms. Mean navigation success: 94.2% versus 61.8% for MOSU; mean condition-detection accuracy: 91.4% versus 72.2%.
Signalized crosswalks go from 42% to 100% success, shared roads from 56% to 100%, construction sites from 50% to 100%, stop gestures from 61% to 71%.
Cost: mean decision latency 3.83 s versus 1.72 s, up to 5.16 s when stage two runs, and about 4.3 times as many input tokens.

Limitations

The number of real-robot trials per group is limited, so the authors call the differences trends. The map suffers from occlusion, segmentation errors and wrong frame-to-frame associations, which corrupt velocity estimates. A cue missed by stage one never reaches stage two, and with decisions taking up to 5.16 s the scene can change before the robot acts.

Videos

Real-world runs: responding to a pedestrian gesture and yielding to a car.

Figures

From Figure 1: MOSU2 supports long-range outdoor navigation across wheeled-legged, wheeled, and legged robot platform... — From Figure 1: MOSU2 supports long-range outdoor navigation across wheeled-legged, wheeled, and legged robot platforms (bottom row). The satellite view illustrates a route from the start (yellow star) to the goal (green circle). The left-side examples highlight traversability challenges; the right-side examples highlight social and traffic challenges such as pedestrians, traffic signals and conflicting traffic cues.

From Figure 2: Overview of the MOSU2 architecture. GPS-based routing provides subgoals, while LiDAR and robot-velocit... — From Figure 2: Overview of the MOSU2 architecture. GPS-based routing provides subgoals, while LiDAR and robot-velocity observations support candidate trajectory generation. RGB-based semantic segmentation provides complementary traversability assessments. The first stage screens the current scene and determines if more analysis is required, and the second stage, when invoked, combines recent images with retrieved geometric and dynamic information to select a trajectory and velocity mode.

From Figure 4: Construction of the VLM-retrievable local map. A VLM generates robot-specific semantic queries for the... — From Figure 4: Construction of the VLM-retrievable local map. A VLM generates robot-specific semantic queries for the segmentation of consecutive RGB images. The corresponding LiDAR points are then used to construct a 2D local map that provides the geometric information of the semantic objects and to calculate the motion information of the objects.

From Figure 10: Qualitative evaluation of MOSU2, external observer view of one of seven representative interactions: ... — From Figure 10: Qualitative evaluation of MOSU2, external observer view of one of seven representative interactions: the robot at a signalized crosswalk, with the traffic lights highlighted.

Abstract

Robots that travel long distances through public outdoor spaces must do more than avoid obstacles: they must remain on terrain suited to their mobility and decide when to proceed, yield, or detour around pedestrians, cyclists, vehicles, traffic signals, and temporary restrictions. These decisions are difficult because their correctness often depends on how a scene evolves, such as whether an approaching car is slowing or a cyclist will cross the robot's path. Vision-language models (VLMs) can interpret such cues, but they reason over images that lack metric geometry and motion. Supplying this evidence at every step is too slow for real-time navigation, and hand-crafted triggers cannot anticipate the open-ended variety of outdoor situations. We present MOSU2, a long-range navigation system with two-stage trajectory selection, in which first-stage contextual reasoning determines when additional evidence is needed and what to retrieve. Only when an interaction remains unresolved does the first stage trigger the second stage to retrieve geometric and dynamic information and align it with scene context to make navigation decisions. Building on MOSU, MOSU2 follows routes from GPS-based route planning and filters candidate trajectories through multi-modal traversability analysis before selecting a trajectory to move the robot. In real-world experiments spanning stop gestures, construction sites, crosswalks, and roads shared with vehicles, cyclists, and pedestrians, MOSU2 raises mean navigation success from 61.8% to 94.2% and condition-detection accuracy from 72.2% to 91.4% over its single-image predecessor, at about 2.2 times its mean decision latency. Project website: https://jinggm.github.io/mosu2/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong