TAMPAgent: Agentic Belief-Space Planning for Mobile Manipulation

Utkarsh A. Mishra, Alperen Degirmenci, Yongxin Chen, Danfei Xu, Tucker Hermans, Caelan Garrett

November 2026

Simulated RB-Y1 trials, sped up, with the plan and its certified steps shown on the right.

TL;DR A language-model agent writes belief-space task-and-motion plans, and perception, grasp and motion tools must certify each action's constraints before the robot moves. On simulated kitchen tasks with an RB-Y1 mobile manipulator it reaches 77.2% success, against 57.2% for Code as Policies and 61.1% for Graph as Policy.

What they do

Agents that write code calling robot tools (Code as Policies, CaP) check no geometric constraints before acting; task and motion planning (TAMP) checks them but needs hand-written problems and fully observed scenes. In TAMPAgent a language-model agent writes the problem and plan, and the planner states which constraints tools must certify before anything moves.

How it works

Hybrid beliefs. Each plan variable (object location, grasp, placement) is a uniform prior, a language hypothesis written by the agent (“behind the penguin”), or a value certified by a tool: a SAM 3 mask, an RGB-D estimate, a GraspGenX grasp or a cuRobo trajectory.
Plan, compile, certify. The planner compiles a plan of detect, pick and place actions into a constraint satisfaction problem. The agent writes a Python graph of tool calls to bind the first action’s parameters, which runs only once every constraint is certified.
Replan and escalate. After each action the agent rewrites the problem. When certification fails, it tries other candidates or viewpoints, then another plan, then new hypotheses, for example moving a blocking object first.

Results

Simulated RB-Y1, 20 episodes per task: 77.2% absolute success, against 57.2% for CaP and 61.1% for Graph as Policy (GaP).
On tasks with a blocker: 75% success, against 50% (CaP) and 56% (GaP). With the apple hidden in the fridge or behind a box, it finds the apple in 14 and 13 of 20 episodes; both baselines manage 9 and 10.
Per simulated minute it uses 19.3k uncached input and 1.75k output tokens, against 20.3k/1.91k for CaP and 27.4k/2.73k for GaP.
Simulated G1, six tasks: 53.3% absolute success, against 6.7% for CaP. On real robots, three RB-Y1 tasks and one G1 blocker-clearing run are shown as demos.

Limitations

The guarantees are only as strong as the tools and perception. Without calibrated probabilities, the agent sometimes keeps searching when acting would succeed, which causes most of its timeouts. The action domain is fixed, and certification protects geometric feasibility but not how the agent interprets the task. The real-robot runs report token use, not success rates.

Videos

In simulation, the agent sees a cup blocking the apple and adds a step to move the cup first.

The same blocked-apple task on a simulated Unitree G1 humanoid.

Figures

From Figure 1: Four ways to map observations to control. Classical TAMP has a task planner orchestrate tools, VLM-TAM... — From Figure 1: Four ways to map observations to control. Classical TAMP has a task planner orchestrate tools, VLM-TAMP lets the agent write the problem but keeps the planner pipeline fixed, Code as Policies lets the agent call tools directly, and TAMPAgent lets the agent orchestrate both the task planner and the tools.

From Figure 4: The planning loop on a real RB-Y1, for an apple hidden behind a toy penguin. The agent observes, guess... — From Figure 4: The planning loop on a real RB-Y1, for an apple hidden behind a toy penguin. The agent observes, guesses where the apple is and writes a plan; the planner compiles it into constraints; tools certify the first action, the robot executes it, and the loop repeats until the apple is in the bowl.

From Figure 7: Absolute and partial success on the simulated RB-Y1, 20 episodes per task and method, grouped into pla... — From Figure 7: Absolute and partial success on the simulated RB-Y1, 20 episodes per task and method, grouped into plain mobile manipulation, tasks with a blocker to move, and tasks with a hidden apple.

From Figure 11: Clearing a blocker on a real G1. The grasp on a toy dog is not certified from the first stance, so th... — From Figure 11: Clearing a blocker on a real G1. The grasp on a toy dog is not certified from the first stance, so the agent adds an approach step; from the new stance the tools certify the grasp and the robot lifts the dog. The run ends before the can is picked.

Abstract

Open-world robotic mobile manipulation requires acting in physical environments with only partial information. While recent Large Vision-Language Model (LVLM) agents show strong open-world reasoning capabilities, existing architectures that rely on direct tool calling struggle with geometric uncertainty and thus are limited to fully-observable tabletop environments. In contrast, Task and Motion Planning (TAMP) systems can solve long-horizon tasks under partial observability but incur high computational costs and require closed worlds. We propose \textbf{TAMPAgent}, an agentic planning framework that unifies the reasoning of LVLMs with the structured constraint satisfaction of TAMP systems. We endow an LVLM agent with a belief-space planning model operating over an extensible API of perception, planning, and control tools. The agent formulates a partial task plan composed of parameterized observation and execution actions, which is subsequently completed using classical search algorithms. After extracting the plan's geometric constraints, the agent leverages its tool API to bind valid parameter values, satisfying feasibility requirements and producing an executable motion plan. Upon execution, the agent continuously updates its belief history and repeats this closed-loop process by interleaving observation and manipulation actions. Evaluated across complex household mobile manipulation tasks on Rainbow Y1 and Unitree G1 platforms, TAMPAgent significantly outperforms direct tool-calling agentic baselines, demonstrating that integrating TAMP constraint satisfaction into agentic planning loops is essential for unstructured, partially observable environments.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong