Bridging Task Planning and Execution for Language-Conditioned Bimanual Manipulation

Emanuele Cuzzocrea, Giuseppina Iannotti, Luca Rossini, Arturo Laurenzi, Nikos Tsagarakis

November 2026

The nine tabletop scenarios on the CENTAURO robot, sped up.

TL;DR A zero-shot VLM planner for a bimanual humanoid that annotates each object side with accessibility and reachability, schedules compatible actions on both arms, and validates every stage. On the real CENTAURO robot it clears side-by-side clutter at 50-85% success where a planner without that information scores 0%.

What they do

VLM task planners reason about what to do but usually leave implicit how each action can physically be carried out, so a box wedged between two others looks graspable to them. This zero-shot framework (no training, fine-tuning or demonstrations) makes those choices explicit for a bimanual humanoid: which side of an object is free, which arm can reach it, and what must hold before and after each step.

How it works

Scene graph. A VLM lists objects and relations from one RGB image; FoundationPose with CAD models gives 6D poses. Each of an object’s six sides is marked accessible (enough clearance for the end-effector, with the blocker recorded otherwise) and reachable (an IK solution exists).
Planning. A VLM outputs actions as (grasp/place/push/release, target, left/right/both arms, contact type, side), relocating blocking objects when no side is free.
Scheduling. Compatible actions are grouped into parallel two-arm stages, each with a local goal and atomic pre- and post-conditions.
Validation. A VLM rates each condition satisfied, uncertain or violated. Uncertain triggers re-perception; a failure is handled by repeat, modify (change arm, contact or side) or full replan, in that order.

Results

Real CENTAURO robot, nine tabletop scenarios, 20 runs each, Gemini Robotics-ER 1.6 as the VLM.
Boxes Aside, Fragile Box and Long Horizon: 85%, 70% and 50% success, against 0% for a planner given only symbolic relations. Push Failure rises from 15% to 95%, Bottles on Box from 70% to 95%.
Hierarchical validation mainly saves queries: a recovery decision costs 1.1 queries in Bottle Failure and 2.1 in Push Failure, against 3 for always replanning.
Parallel scheduling cuts robot motion time by 40% in Bottles on Box and 11% in Long Horizon.
Swapping the VLM: GPT-o3 reaches 80.0% at 91.4 s latency, Gemini 73.3% at 41.9 s, GPT-5.2 5.0%.

Limitations

Objects are oriented boxes grasped on opposing faces, so handles and other instance-specific grasps are out of scope, and poses need CAD models. Motions come from inverse kinematics with a fixed base; unreachable sides are discarded. Long-horizon planning is the weak module (67% correct), and perception suffers from the head camera’s viewpoint.

Videos

Stacked Boxes at 5x: the generated plan is shown next to the robot as it picks up the white box.

The three failure scenarios side by side: recovery by repeat, modify and replanning.

Figures

From Figure 1: Execution-aware target retrieval in clutter. The framework builds a structured scene representation, p... — From Figure 1: Execution-aware target retrieval in clutter. The framework builds a structured scene representation, plans the manipulation actions and their physical realisation, and schedules compatible actions for parallel bimanual execution. A hierarchical validator monitors each execution stage.

From Figure 2: Overview of the pipeline. From an RGB observation and object poses and dimensions, the framework const... — From Figure 2: Overview of the pipeline. From an RGB observation and object poses and dimensions, the framework constructs a structured scene representation, generates a sequential manipulation plan, and groups compatible actions into parallel execution stages. A hierarchical validator evaluates stage-level pre- and post-conditions and guides a local-to-global recovery process.

From Figure 4: Example execution of the Fragile Box scenario, where the task is to grasp the white box. The robot fir... — From Figure 4: Example execution of the Fragile Box scenario, where the task is to grasp the white box. The robot first pushes the non-fragile blue box from its accessible front side, then grasps and places the fragile green box on the table, and finally grasps the white box.

Abstract

Robots are increasingly expected to execute natural-language manipulation tasks in real-world environments. This is particularly relevant in household assistance and logistics, where clutter requires reasoning about both task objectives and execution feasibility. This challenge becomes even more pronounced for bimanual humanoid platforms, where coordinating two manipulators introduces additional interaction and kinematic constraints. We present a zero-shot vision-language framework for execution-aware bimanual manipulation that jointly reasons about task objectives and physical feasibility. Planning is grounded in a structured scene representation that combines semantic object relations with execution constraints, enabling the selection of obstacle relocations and execution choices while preserving task semantics. Compatible actions are organised into parallel bimanual stages with explicit pre- and post-conditions. A hierarchical validator evaluates these conditions under perceptual uncertainty and uses stage-level execution history to support local recovery or targeted replanning. Experiments on the CENTAURO hybrid wheeled-legged humanoid robot across nine language-conditioned manipulation scenarios show higher task success than a non-execution-aware baseline. In dedicated failure scenarios, hierarchical validation recovers from execution failures with fewer model queries than always replanning. These results demonstrate the benefits of combining execution reasoning directly into task planning. Videos and additional material are available on the Project Webpage: https://advrhumanoids.github.io/bridging-planning-execution/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong