Do Agentic Policies Need to Look at Every Step? Geometric Abstraction for Reasoning and Action

Yihang Liu, Yiming Chen, Mingzhe Zheng

November 2026

A synthesized policy picks and places blocks for the Bridge+cylinder task in simulation.

TL;DR Instead of showing an LLM agent camera images at every step, they give it the scene's 3D geometry as text (object faces, edges and goal residuals) grounded locally from RGB-D. On simulated SO-100 tasks this raises online tower-relocation completion from 30% to 84% while using about half the tokens.

What they do

LLM agents that drive manipulation usually re-read camera images at every decision. This paper instead gives the model an explicit 3D geometric state in text, refreshed locally from calibrated RGB-D, and lets it ask for images only when needed. It is compared with a Pixel Reasoning agent that has the same task, sensors, skills and model, on a simulated SO-100 arm in MuJoCo.

How it works

Grounding. Side and wrist depth is back-projected into the world frame; color segmentation and CPU cuboid/cylinder fits give each object’s center, vertices, edges and oriented faces. Every element is tagged observed, inferred from the shape prior, or unavailable. After an object moves, its pose must be re-measured, never copied from the commanded motion.
Symbols and operators. Policies refer to parts like red.bottom_face and call operators such as align (rigid transform mating two faces) and support checks that return metric residuals and true/false/unknown.
Textual checkpoints. After each action the agent gets updated coordinates, the goal residual and its evidence status.
Two regimes. Policy synthesis: the LLM revises a Python policy over three rounds on five layouts, then it is tested on 100 held-out layouts. Online: the LLM picks skills in one non-resetting 200 s episode while physics keeps running.

Results

Policy synthesis, after the first revision: 45.2% vs. 21.2% held-out success on Bridge+cylinder and 57.2% vs. 17.2% on Three-cube stacking; final 65.4% vs. 56.0% and 81.4% vs. 70.8%. Ring-on-peg reaches 38.0% vs. 0.0%.
Online: Bridge-tower 28/50 vs. 12/50, Tower relocation 42/50 vs. 15/50. Ring-on-peg succeeds 50/50 vs. 49/50 but finishes 33.3% faster.
On Ring-on-peg, model-request time drops 49.4%, tokens 52.8%, and attached images per episode from 10.94 to 2.72. Bridge-tower uses 15.1% more tokens.

Limitations

Simulation only, with regular table-top objects; real sensor noise, heavy occlusion and calibration drift are untested. Perception relies on color segmentation and hand-written shape estimators, and articulated or deformable objects are left for future work.

Videos

Online execution of Bridge-tower in one continuing episode without resets.

Online execution of Tower relocation, moving the top cubes of a column onto a new foundation.

Figures

From Figure 1: Visual observations are grounded locally into metric 3D geometry and passed to the language model as s... — From Figure 1: Visual observations are grounded locally into metric 3D geometry and passed to the language model as symbolic references and compact textual checkpoints, with images sent only on demand.

From Figure 2: RGB-D observations are grounded into object geometry (points, edges and faces) and exposed to the LLM ... — From Figure 2: RGB-D observations are grounded into object geometry (points, edges and faces) and exposed to the LLM through typed symbolic references and textual checkpoints. In policy synthesis the LLM revises an executable policy between rollouts; in online execution the same interface supplies updated geometry for each decision within one continuing episode.

From Figure 6: Online completion rate within 200 s, mean capped completion time and mean recorded tokens over 50 pair... — From Figure 6: Online completion rate within 200 s, mean capped completion time and mean recorded tokens over 50 paired layouts per task, for Pixel Reasoning and Geometric Abstraction.

Abstract

Recent multimodal language models can reason directly about robot scenes, but it is unclear whether they need to repeatedly reinterpret raw visual observations at every step of an interaction. We propose shifting routine spatial reasoning from images to text and symbols through a 3D geometric abstraction that is refreshed locally from observations and read directly by the language models. Specifically, our interface represents scene geometry via typed, evidence-aware symbolic references to object parts, grounds them from calibrated RGB-D, and returns compact textual checkpoints with metric residuals after each action. Symbolic task structure can thus be reused across decisions while metric values are rebound to the current scene, with direct visual reasoning invoked only on demand. Across offline policy synthesis and continuous online execution, reasoning over grounded text and symbols improves multi-step manipulation completion while substantially reducing repeated multimodal queries, positioning explicit geometric state as an effective substrate for agentic robot policies.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong