← Paper Digest

InteractVLA: Turning VLAs into Interactive Tools for Agentic Robotic Manipulation

Chen Zhao, Zipeng Ye, Shifeng Bao, Yihan Lin, Haitao Shen, Yining Luo, Yuchen Xie, Jing Zhang

November 2026

TL;DR InteractVLA lets a vision-language agent steer a frozen flow-based VLA toward an end-effector target pose by guiding its denoising, and read back the planned motion to correct it. It reaches 99.4% on LIBERO-Plus and, on four real bimanual tasks, roughly doubles task scores over a language-only agent on pouring and bottle uprighting.

What they do

Agent systems usually call a pretrained VLA with a language instruction only. When a call fails, a retry gives nearly the same motion, and rephrasing pushes the VLA out of distribution. InteractVLA gives the agent a geometric interface instead: it passes the VLA an end-effector target (position, orientation, gripper state) and reads back the action chunk the VLA generates. The VLA is not retrained.

How it works

  • Test-time geometric guidance. A differentiable map from actions to end-effector poses lets the pose error be backpropagated through the action expert during flow-matching denoising, steering the chunk toward the target. If a confidence score is low after the first guided step, generation stops and the agent replans.
  • Soft-time target alignment. The arrival step is not fixed: a log-sum-exp over per-step pose errors spreads guidance over compatible steps and sharpens onto one as denoising proceeds, so the approach motion is preserved.
  • Action-informed refinement. The agent sees the generated chunk and its predicted poses reprojected into the calibrated camera, then either revises the target and calls the VLA again or edits the motion locally (approach offset, lift height, release timing). Over practice episodes in scenes kept separate from evaluation, the agent also accumulates strategies (recursive self-improvement, RSI).

Results

  • LIBERO-Plus, online tool use (MR-0 VLA, GPT-6.1 Sol agent): 93.1% with language-only calls to 98.3%; under layout shifts 89.5% to 98.4%. With RSI: 99.4%.
  • RoboDojo: 18.0% to 26.0% online; with RSI and GPT-6 Astra, 48.62% against 46.85% for the best listed agent. RoboCasa365 with RSI: 89.4% against 55.0% for the base policy.
  • Real Dexmal robot, 30 trials per task: task score against a language-only agent rises from 65.5 to 92.5 (tidying), 37.9 to 74.6 (upright bottle), 38.5 to 79.2 (pouring) and 86.7 to 97.6 (move cube, with time down from 165.2 s to 67.5 s).
  • Soft-time alignment beats uniform (34.7%) and fixed-final-step (39.1%) alignment on RoboDojo (48.6%).

Limitations

The paper has no limitations section. On RoboDojo precision tasks, online tool use drops slightly (9.7% to 8.3%). With an agent in the loop, real tasks take much longer than the standalone VLA (295.8 s against 97.4 s on tidying). The method needs a flow-based VLA, calibrated cameras and a commercial LLM agent.

Figures

From Figure 2: (a) Matching a sparse target too early disrupts the approach, causing early gripper closure and rotati...
From Figure 2: (a) Matching a sparse target too early disrupts the approach, causing early gripper closure and rotation; timely alignment preserves approach, contact and lift. (b) InteractVLA steers a frozen VLA toward an agent-specified target and returns action feedback to the agent. (c) Online tool-use success on LIBERO-Plus and RoboDojo with and without InteractVLA.
From Figure 3: Left: at each denoising step, target similarity sets the guidance weights across the action chunk, ste...
From Figure 3: Left: at each denoising step, target similarity sets the guidance weights across the action chunk, steering the generated motion toward the agent-specified target. Right: different targets give different motion proposals that the agent reprojects, compares and locally refines; low-confidence targets are rejected and replanned.
From Figure 6: Real-world runs on the Dexmal platform, top to bottom: Tabletop Tidying, Upright Bottle, Water Pouring...
From Figure 6: Real-world runs on the Dexmal platform, top to bottom: Tabletop Tidying, Upright Bottle, Water Pouring and Move Cube, each row showing successive stages from left to right. Move Cube is shown from the overhead camera to show the tight clearance between cube and cup.
Abstract

Existing agentic manipulation systems treat pretrained vision-language-action (VLA) policies as opaque, language-only primitives: when a policy call fails under unfamiliar instructions or scene perturbations, the agent cannot steer its behavior and is forced into futile retries or costly manual takeover. We present InteractVLA, a framework that turns pretrained flow-based VLAs into geometrically steerable and inspectable tools through a bidirectional geometric interface with vision-language agents. In the forward direction, test-time geometric guidance translates an agent-specified end-effector target into input gradients through the action expert via a differentiable command-to-pose mapping, while soft-time target alignment treats arrival time as a latent execution index to match the target pose without distorting the policy's approach motion. In the reverse direction, action-informed agentic refinement exposes generated action chunks and calibrated camera reprojections back to the agent, allowing it to diagnose spatial discrepancies, revise targets for subsequent VLA calls, or locally adjust motion segments on top of the guided trajectory. Making the VLA controllable not only boosts online agentic tool use, but also unlocks effective recursive self-improvement (RSI), achieving state-of-the-art performance of 99.4\% on LIBERO-Plus, 48.62\% on RoboDojo, and 89.4\% on RoboCasa365. Project page: https://lingwupro.github.io/InteractVLA/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.