SNAP: Semantic Navigation Among Pedestrians

Jakob Wolfram, Pascal Roth, Fan Yang, Cesar Cadena, Marco Hutter

November 2026

ANYmal-D weaves between pedestrians crossing its path (1.5x speed).

TL;DR An RL policy steers a legged robot through crowds while keeping to safe terrain, using only a semantic bird's-eye-view map with pedestrians drawn in and no pedestrian tracking. It beats crowd-navigation and semantic-planning baselines in simulation (91.8% vs 61.7% success with 10 pedestrians against DRL-VO) and runs zero-shot on ANYmal-D.

What they do

City robots must avoid pedestrians and also follow norms such as staying on sidewalks. Crowd-navigation methods usually need explicit pedestrian positions and velocities, while semantic planners treat the scene as static. SNAP trains a single RL navigation policy on a robot-centric, semantically annotated bird’s-eye-view (BEV) map that holds both terrain classes and nearby moving obstacles, with no pedestrian state estimation. The policy is deployed zero-shot on an ANYmal-D quadruped.

How it works

Training worlds. Procedurally generated terrain of varying risk, static obstacles and pedestrians driven by a randomized Social Force Model. Pedestrians walk norm-respecting paths, sometimes in groups, and sometimes slow down for the robot; a curriculum grows the crowd.
Policy. PPO with an asymmetric actor-critic: the actor sees proprioception, the goal and a 5 x 5 m BEV map (CNN encoder), and only the critic gets pedestrian states. It outputs body-frame velocity commands for a frozen learned locomotion policy.
Rewards. Goal and progress terms; penalties for being near pedestrians, for high-cost terrain under the feet and for failures; smoothness regularizers.
Deployment. 10 Hz on ANYmal-D, localized in a pre-defined semantic map with Open3D SLAM; LiDAR-detected pedestrians are projected into the BEV map.

Results

Success rates over 500 simulated episodes per setting:

Against SARL (5 pedestrians): 93.2% vs 37.4% (original) and 59.2% (retrained) at 0.5 m/s, and 58.6% vs 19.0% and 38.4% at 1.3 m/s.
Against DRL-VO with static obstacles: 91.8% vs 56.6% and 61.7% with 10 pedestrians; 81.8% vs 43.0% and 50.4% with 20.
Against ViPlanner on semantic terrain: 96.0% vs 80.4% without pedestrians, 82.6% vs 58.0% with 16. A front-only 90° BEV variant gets 89.2% and 76.2%.
On ANYmal-D, the robot stays in the safe region, passes between crossing pedestrians and reaches the goal without collision; no real-world success rates are reported.

Limitations

The 5 x 5 m view leaves little time to react to fast pedestrians: with 13 of them, success falls from 84.4% at 0.5 m/s to 45.2% at 1.3 m/s. Deployment uses a pre-defined semantic map instead of onboard semantic perception.

Videos

Real-world run across a hall with low- and high-risk terrain marked (5x speed).

In simulation, the policy heads for its goal past pedestrians and risky terrain.

Figures

From Figure 1: Real-world crowd navigation with ANYmal-D. The policy avoids pedestrians who repeatedly cross its path... — From Figure 1: Real-world crowd navigation with ANYmal-D. The policy avoids pedestrians who repeatedly cross its path while keeping to the semantic structure of the terrain; insets show the robot-centric BEV maps, and the commanded forward velocity (bottom) drops and briefly reverses as a pedestrian approaches.

From Figure 2: (a) Training in procedurally generated environments with semantic terrain of varying risk, static obst... — From Figure 2: (a) Training in procedurally generated environments with semantic terrain of varying risk, static obstacles and SFM-driven pedestrians. (b) The actor observes a robot-centric semantic BEV map, proprioception and the target vector. (c) CNN and MLP encoders feed a Gaussian action head. (d) A frozen locomotion policy tracks the velocity command. (e) PPO training with an asymmetric critic that receives privileged observations.

From Figure 4: Simulation snapshots. (a)-(d) Avoidance maneuvers in the benchmark settings, with past robot positions... — From Figure 4: Simulation snapshots. (a)-(d) Avoidance maneuvers in the benchmark settings, with past robot positions showing paths around crowds and static obstacles. (i)-(iv) The policy passing an oncoming group in a corridor bounded by high-risk terrain; insets show its 5 x 5 m BEV observation (forward is up).

Abstract

Navigating crowded urban environments requires robots to avoid pedestrians while adhering to social norms such as remaining on sidewalks. Existing approaches typically address either crowd avoidance or semantic path planning in isolation. We propose a reinforcement learning (RL) navigation policy that jointly reasons about semantic terrain and dynamic obstacles from a single semantically annotated bird's-eye-view (BEV) map, without relying on explicit human-state estimation. Our policy outperforms established crowd-navigation and semantic-aware baselines and transfers zero-shot to a legged robot, demonstrating socially compliant navigation in highly dynamic real-world scenarios.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong