Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Visibility-Aware Task and Motion Planning with Task-Centric Perception

Sahit Chintalapudi, Xiaoyan Wang, Aryan Zoroufi, Aditya Agarwal, Nancy Kanwisher, Tomás Lozano-Pérez, Leslie Pack Kaelbling

November 2026

TL;DR VA-TAMP wraps a fully observed TAMP solver so the robot must see the space it moves through and the surfaces it grasps before acting, inserting wrist-camera looks and refining only task-relevant object meshes. On three occluded real-robot tabletop tasks it succeeds in 9/10, 10/10 and 10/10 trials, against 2/10, 3/10 and 0/10 for the open-loop TiPToP baseline.

What they do

A task and motion planning (TAMP) solver plans as if it knows every object’s shape, which fails when objects are occluded or only partly in view. VA-TAMP keeps an off-the-shelf, fully observed solver (cuTAMP) but requires every action in its plan to be justified: before the robot moves through space or grasps a surface, it must have observed that space empty and that surface present. Wrist-camera look actions are inserted to gather those observations.

How it works

  • Object-based memory. A VLM (Gemini-ER-2) proposes boxes and SAM2 turns them into masks; detections join object slots by point overlap and DINO feature similarity. An octree occupancy grid labels space unknown, empty or full.
  • Task-centric mesh refinement. Image-to-3D completion (RecGen) takes 4-5 s per object, so each perception step completes one object: the one whose mesh fits its observations worst, weighted by relevance to the current plan. The new mesh is kept only if it explains the observations better.
  • Plan justification. A pick needs the finger-contact points observed full and the arm’s approach volume observed empty; a place needs observed support under the footprint and empty swept volumes. Looks, chosen by a greedy set cover over sampled camera poses, go before each pick; a failed check triggers replanning.

Results

Franka FR3 with a ZED-M wrist camera, 10 real trials per task:

  • Cubes on Bread (the bottom cube is hidden by the top one): VA-TAMP 9/10, TiPToP 2/10.
  • Teacup on Plate (clutter): VA-TAMP 10/10; the visibility-unaware ablation 9/10, but only 3/10 when its mesh refinement is task-agnostic.
  • Supplies on Tray (supplies behind boxes, unseen obstacles on the tray): VA-TAMP 10/10; every other method 2/10 or less, since only VA-TAMP plans looks that do not touch any object.

Limitations

The initial view must contain every object named in the goal, since the nominal plan decides where to look. Looks never move occluders out of the way. The shape model conditions on a single image. The evaluation covers three tabletop scenes.

Figures

From Figure 1: The initial wrist view. The goal is to put the spray bottle and the green eraser on the red tray; the ...
From Figure 1: The initial wrist view. The goal is to put the spray bottle and the green eraser on the red tray; the cardboard boxes partly hide both supplies.
From Figure 3: The visibility precondition of a planned pick on the green eraser, before the look that satisfies it.
From Figure 3: The visibility precondition of a planned pick on the green eraser, before the look that satisfies it.
From Figure 3: The same pick precondition after the look, with the eraser's updated mesh.
From Figure 3: The same pick precondition after the look, with the eraser's updated mesh.
From Figure 4: The Teacup on Plate scene. The robot must take the occluding block off the blue cup, then move the cup...
From Figure 4: The Teacup on Plate scene. The robot must take the occluding block off the blue cup, then move the cup to the flower-shaped plate among many task-irrelevant objects.
Abstract

Long-horizon manipulation under uncertainty remains a challenging problem in robotics. We present VA-TAMP, an algorithm for Visibility-Aware Task and Motion Planning. VA-TAMP extends existing deterministic task and motion planning solvers to partially observable object rearrangement tasks, while still being able to take advantage of the efficiency and compositional generalization over long horizons of the base solver. This extension interleaves sensing actions into a plan skeleton that assumed full observability, and uses a novel persistent object-based memory to fuse observations into object mesh hypotheses and an observed occupancy map. We evaluate VA-TAMP on real-world tabletop rearrangement tasks which feature heavy occlusion and partial visibility and demonstrate that it outperforms baselines that do not perform active perception or have persistent memory.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.