Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

FEIJOA: Task-Driven Grounding and Execution for Mobile Manipulation

Aditya Agarwal, William Shen, Sahit Chintalapudi, Lucy Cai, Nishanth Kumar, Aidan Curtis, Travis Manderson, Tomás Lozano-Pérez, Leslie Pack Kaelbling

November 2026

Spot drives to the whiteboard and wipes a drawing off it.

TL;DR A language-model agent for mobile manipulation that grounds each instruction against a raw RGB-D scan when it arrives, and checks every skill's feasibility before running it and its effect afterwards. It resolves 94.2% of 103 open-ended grounding queries exactly (68.0% for the best prebuilt map) and completes 35 of 50 real tasks on Spot.

What they do

Scene maps for mobile manipulation are built before the instruction is known, so they can miss what a task hinges on, such as “the plush toy with a bow tie”. FEIJOA is a language-model agent (Gemini-3.8-Flash) that grounds each reference against the raw posed RGB-D scan once the instruction arrives, then acts through a harness that checks each skill is feasible and verifies its effect.

How it works

  • Task-driven grounding. A grounding subagent, Fido, turns the instruction into label queries. Its tool runs text-prompted SAM 3 on every scan frame, lifts the masks to 3D with depth and camera poses, merges repeat views into instances and confirms each with Qwen3-VL-8B. The agent inspects the candidates and re-queries until the instruction is resolved.
  • Tools and plan. The agent also calls navigate, pick, place, push and wipe skills, estimates navigation cost, and keeps a task plan of subgoals it updates from execution feedback.
  • Feasibility-aware sampling. Base poses facing the target are sampled on an occupancy grid built from the scan, filtered by footprint collision and the skill’s reach, and ranked by reach margin and travel cost. If none pass, the tool refuses and says why.
  • Effect verification. A separate Gemini-3.6-Flash judge sees only the claimed effect and before/after images, and returns success, failure or uncertain.

Results

  • Grounding, 103 queries over four scans: 94.2% exact match, against 68.0% for a captioned scene graph (the best prebuilt map) and 47.6% for the grounding tool alone, at 29.5 s and 3.1 cents per query. A standalone Gemini 3.8 Flash shown every frame reaches 96.1%, but returns only one point per object.
  • Spot: 35/50 tasks succeed (13/16 short, 12/18 medium, 10/16 long horizon), 17 of them after recovering from a failure. Of the 15 failures, 13 come from robot skills.
  • Ablations: replaying 20 failed place/wipe steps, the agent wrongly marks 9 as done given only status text, 6 with before/after images and 2 with the verifier’s verdict. Without cost estimation it drives 9.8% further and takes 12.3% longer.

Limitations

The robot’s physical skills are the main bottleneck. Query-time grounding is slower than looking up a prebuilt map, and language-model calls take 41% of wall time (about 20 per trial at 7 s each). The method needs a prior scan and assumes the scene changes only through the robot’s actions. All 50 trials are on Spot.

Videos

Spot moves plush toys from a table into a bin.
Spot on a button-pushing task, with the buttons on the floor.

Figures

From Figure 1: FEIJOA on Spot, told to
From Figure 1: FEIJOA on Spot, told to "erase whatever is on the whiteboard that's bringing down the mood, then put the thing you used to erase back on the table next to the burger". Left to right: it picks up the mitt, drives to a base pose with the board in reach, wipes the sad face away, checks the before and after images, and returns the mitt beside the toy burger.
From Figure 2: FEIJOA overview. Each turn, the harness builds the working context from a fixed context, the interacti...
From Figure 2: FEIJOA overview. Each turn, the harness builds the working context from a fixed context, the interaction history, the current observations and the task plan; the language model picks tool calls and the harness executes them. Fido, the grounding subagent, finds object hypotheses in the recorded scan, and every skill goes through feasibility-aware sampling, precondition checks and effect verification.
From Figure 4: Task-conditioned grounding. For a query such as
From Figure 4: Task-conditioned grounding. For a query such as "plush toy", the grounding tool detects candidates in every scan frame, lifts them to 3D, clusters repeat views into instances and has a VLM verify each one, then returns their ids, positions, extents and best views to the agent.
From Figure 3: Outcomes of 50 Spot trials. 17 of the 35 successes recovered from failed tool calls or verifier checks...
From Figure 3: Outcomes of 50 Spot trials. 17 of the 35 successes recovered from failed tool calls or verifier checks, mostly by re-navigating and retrying. Almost every failure is a robot skill going wrong; only two are due to grounding or the agent's reasoning.
Abstract

A mobile manipulator asked to "put the toys away in the toy box" must decide which objects are toys and which container the user means, then find base poses, grasps, and placements it can execute. Because a scene representation is built before the instruction is known, it may lack the attributes, affordances, or concepts needed to resolve the task. We present FEIJOA, a language-model agent that grounds the instruction at test time. It queries a grounding tool over a posed RGB-D scan, inspects the returned candidates, and refines its queries until they resolve the instruction. An embodiment-aware harness checks each skill's preconditions, samples base poses from which the target is reachable, and verifies the skill's effect from observations before and after execution. On 103 open-ended grounding queries, FEIJOA resolves 94.2% exactly, compared with 68.0% for the best prebuilt representation. Over 50 real-world trials on Spot, it completes 35 tasks; 17 of these successes follow recovery from a failed tool call or verification check. Of the 15 failed trials, 13 were caused by the robot's physical skills, and only 2 by grounding or the agent's reasoning. Project website: https://williamshen-nz.github.io/feijoa/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.