Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

MessyNav: Zero-Shot Navigation via Manipulation in Messy Environments

Kenneth Llontop, Cherie Ho, Dylan Zhou, Anuva Banwasi, Priya Sundaresan, Carlota Parés-Morlans, Jeannette Bohg

November 2026

Annotated bedroom run where the book-stacked suitcase is labeled non-interactive and the planner swipes the cart.

TL;DR MessyNav lets a mobile manipulator clear blocked paths zero-shot. A VLM first decides from context which obstacles are safe to move, then a VLM planner sequences swipe, pick-up and navigation primitives. On a real TidyBot++ it succeeds in 67% of messy-room trials versus 40% for end-to-end VLM planning.

What they do

Standard navigation stacks treat every obstacle as a wall. MessyNav lets a mobile manipulator decide, zero-shot, whether a blocking object should be swiped aside, picked up or avoided, based on the VLM’s reading of its context. An empty cart can be pushed, but a cart with a laptop on it cannot.

How it works

  • Affordance reasoner. A VLM (gemini-3-flash-preview) looks at the front camera image and labels each object interactive or non-interactive. It first checks what the robot can physically move (payload, skills), then whether moving the object is safe (fragile, valuable, deliberately arranged).
  • Task planner. A second VLM call takes the labels and the image and outputs a plan, e.g. move_to, swipe_right, move_to.
  • Skill library. Object names are grounded in 3D with VLM pointing, segmentation and depth. move_to uses Nav2 with LiDAR; pick_up and lateral swipes are open-loop arm primitives on a TidyBot++ (Kinova arm, holonomic base).

Results

  • Affordance benchmark (100 images, 217 objects): 85.7% accuracy vs 66.4% for a closed-set oracle with fixed category labels and 59.9% for a height heuristic. On context-dependent cases it gets 85/105 vs 38/105 for the oracle.
  • Real robot, 15 scenes in a bedroom and a conference room: 10/15 successes. Planning end to end with the VLM, without the affordance stage, reaches 40%. Planning failures drop from 7 to 2; most remaining failures are wrong VLM pointing.
  • Offline planning over 55 scenes and 15 runs: 62.4% vs 55.4% (p<0.001), but 29.2% vs 44.2% at night.

Limitations

Primitives run open-loop, so fragile objects are avoided rather than handled gently. Decisions are bounded by the VLM, which degrades in poor lighting (worse than the baseline at night) and when it has to reason about geometry from 2D images. Without persistent memory, the robot cannot learn from a failed interaction. The real-robot comparison uses one trial per scene.

Videos

Outdoor daytime run on a campus patio.
Outdoor run at night beside a bike rack.

Figures

From Figure 1: Top: MessyNav labels the utility cart carrying a plastic bin as swipeable and the crate carrying unsta...
From Figure 1: Top: MessyNav labels the utility cart carrying a plastic bin as swipeable and the crate carrying unstable stacked blocks as non-interactive, then swipes the cart aside to reach the goal. Bottom: deployments across indoor and outdoor environments.
From Figure 2: A VLM Affordance Reasoner labels object interactability from an RGB image, checking physical feasibili...
From Figure 2: A VLM Affordance Reasoner labels object interactability from an RGB image, checking physical feasibility and interaction safety. A Task Planner then sequences actions from a skill library (pick_up, swipe_left, swipe_right and LiDAR-based move_to), and object references are grounded in 3D with VLM keypoints, segmentation and depth.
From Figure 4: A suitcase supporting a stack of books is unstable, while a cart holding one book can be cleared. Mess...
From Figure 4: A suitcase supporting a stack of books is unstable, while a cart holding one book can be cleared. MessyNav labels both correctly and swipes the cart; vlm-planner-only and closed-set-planner both choose to swipe the suitcase.
Abstract

Real-world environments are messy, with paths often blocked by everyday objects like chairs and toys, requiring robots to perform interactive navigation by manipulating obstacles to clear a path. Existing methods rely on extensive simulation-based learning or per-scene instructions specifying which objects to avoid or manipulate. They struggle to generalize across novel objects, diverse environments, and varied contexts such as a cart of fragile glassware that should be avoided while an empty one can be pushed. We formalize this problem as open-world interactive navigation and present MessyNav, a zero-shot framework based on vision-language models (VLMs). At its core, MessyNav performs explicit VLM-based affordance reasoning before task planning to decide whether to swipe, pick up, or avoid an object and produce a high-level plan. Evaluated on a mobile manipulator across real-world indoor and outdoor environments, MessyNav improves success rate by 27 percentage points (pp) over the strongest baseline in real-robot trials and achieves statistically significant gains in a larger-scale offline evaluation, without task-specific training.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.