Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

WayTrace: A Map-to-Language-to-Trace Hierarchy for Embodied Urban Navigation

Manthan Patel, Jonas Frey, Pascal Roth, Marco Hutter, Cesar Cadena

November 2026

Real-world mission to the nearest Starbucks, with the robot camera, predicted trace and route overview (2x speed).

TL;DR WayTrace turns web-map walking directions into one language command at a time, draws each command as a path in the camera image with a fine-tuned VLM, and has an RL policy follow it. It completes 83.3% of ten simulated Zurich missions (no baseline exceeds 20%) and real city missions of up to 1.37 km on a wheeled-legged robot.

What they do

Web maps give walking directions for almost any city, but point-goal navigation policies depend on accurate GNSS, and vision-language navigation models are brittle even on single instructions. WayTrace splits long-range urban navigation into two problems: deciding which atomic manoeuvre applies now (“cross the road to the left”), and executing it from the camera image. Language connects the two. The system runs on a Unitree B2W wheeled-legged robot.

How it works

  • Orchestrator. Every 2 s, an off-the-shelf VLM on a remote server reads the web-map directions, a 50 m satellite map and camera images of the last 15 m, then issues the next command. GNSS only selects the current route step.
  • Trace predictor. RynnBrain-9B, fine-tuned with LoRA, takes the command, five past frames and the walked path, and predicts a goal point and a ten-point path in the image. It runs onboard on a Jetson AGX Thor in about 1.5 s.
  • Training data. 1.19M samples from street-level image sequences: 288k from reconstructed walks labelled by a VLM, and 863k intent-conditioned traces that pair one image with several commands.
  • Execution. Stereo depth lifts the trace to 3D, and an RL point-goal policy follows a carrot 4.5 m ahead at 5 Hz while avoiding obstacles.

Results

  • Ten simulated Zurich missions (4.68 km in total) with perfect GNSS: 83.3% success, against at most 20% for released point-goal and vision-language baselines.
  • With 7 m RMS GNSS drift, success falls to 63.3%, against 23.3% and 10.0% for CityWalker and VAMOS on the same RL policy. Crosswalk and road violations stay below 6% and 1%.
  • Open-loop trace prediction scores 83.2 (NaviTrace score) on hand-labelled traces, against at most 57.0 for baselines.
  • Real world: three city-centre missions of 248-426 m with no interventions, and a 1.37 km mission with two operator overrides of the current command.

Limitations

Most failures come from the orchestrator, which issues turns or crosswalk entries too early or too late (5.1% critical errors per decision). About 3 s of latency rules out reacting to red lights or trams. The simulation has no traffic or pedestrians, and the real-world missions are qualitative demonstrations without baselines.

Figures

From Figure 2: One logged decision from the real-world Snipes mission. (A) The web map gives the route and turn-by-tu...
From Figure 2: One logged decision from the real-world Snipes mission. (A) The web map gives the route and turn-by-turn directions; GNSS only picks the next step. (B) Every 2 s, the orchestrator VLM reads the directions, a satellite map and recent camera images and issues the next atomic command. (C) Onboard, the trace predictor uses the command, past frames and the walked path to predict a goal point and a ten-point trace in the image. (D) The trace is lifted to 3D with stereo depth and an RL policy follows a carrot 4.5 m ahead. The timeline at the bottom shows the loops running asynchronously.
From Figure 3: Training data of the trace predictor. Top: reconstructed walks, where a street-level sequence is recon...
From Figure 3: Training data of the trace predictor. Top: reconstructed walks, where a street-level sequence is reconstructed, the walked path is projected into every frame, and VLMs write and check the matching command. Bottom: intent-conditioned traces, where a teacher draws one trace per VLM-drafted command on an anchor image and the traces are propagated to later frames. Right: composition of the 1.19M training samples.
From Figure 5: Long-range navigation under GNSS noise. Success rate, crosswalk violation and road violation of WayTra...
From Figure 5: Long-range navigation under GNSS noise. Success rate, crosswalk violation and road violation of WayTrace and of point-goal baselines driven by the same RL policy. The dotted line marks WayTrace with an oracle orchestrator at perfect GNSS.
From Figure 6: Three real-world missions in Zurich. Left: planned route, driven path, orchestrator commands and opera...
From Figure 6: Three real-world missions in Zurich. Left: planned route, driven path, orchestrator commands and operator overrides. Right: three moments per mission with the robot camera, the predicted trace and the executed command.
Abstract

Web services provide walking directions for nearly every city, but following them reliably over long routes remains challenging for autonomous robots. Point-goal navigation policies rely heavily on accurate GNSS, while most vision-language navigation models struggle to execute even short instructions outdoors. We present WayTrace, a hierarchical system that connects web-map directions to robot motion through language. An off-the-shelf orchestrator vision-language model reads the turn-by-turn directions, a satellite map and recent camera images, and issues one atomic language command at a time, such as "cross the road to the left". An onboard trace predictor, trained on over one million automatically labelled samples from street-level images, draws the path for this command in the image, and a reinforcement learning policy tracks it. In long-range simulated missions, WayTrace remains robust to GNSS noise and rarely leaves crosswalks or sidewalks, while point-goal baselines degrade sharply with about ten times more violations. Deployed on a wheeled-legged robot, WayTrace completes real-world missions of up to 1.37 km at about 1.1 m/s through a busy city centre, rural trails and a university campus, handling crosswalks, staircases and construction sites. Project page: https://way-trace.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.