← Paper Digest

A$^2$-VLA: Adapting Vision-Language-Action Policies to New Tools through Agentic Affordance Discovery

Chenyuan Zhou, Chaoyi Huang, Zibo Wang, Zhiyuan Li, Yong-Lu Li

November 2026

TL;DR An LLM agent rebuilds tool-use demonstrations in simulation, works out which part of a tool does the job (a hook, an edge, a support surface), and feeds those features to a π0.5 policy. On a real robot with unseen tools, success rises from 4.7% for plain π0.5 to 35.3%.

What they do

A VLA policy trained on demonstrations with a few tools tends to fail when handed a differently shaped one. In A2-VLA an LLM agent (GPT-6 Astra) works out which part of each tool does the job, such as a hook, a cutting edge or a support surface, tests that hypothesis in simulation, and passes the resulting geometric features to a π0.5 policy. A new tool then needs only an RGB-D inspection, not new demonstrations.

How it works

  • Real-to-sim. The agent rebuilds each teleoperated demonstration (scene, tools, recorded robot motion) in simulation.
  • Affordance discovery. Comparing source tools, it picks a functional region and its geometry (position, tangent, opening or cutting direction) and writes code to extract them. It tests these features by retargeting a demonstration to other tools in simulation and revises regions, features or code from the outcome. Validated extraction skills go into a reusable library.
  • Affordance-conditioned VLA. The 12-D feature vector becomes one extra token for π0.5, an auxiliary head predicts the future affordance trajectory, and 10% of the training data are counterfactual retargeted trajectories so the policy learns to follow the features.
  • New tools. The agent reconstructs the unseen tool, extracts and refines its features in simulation, and the VLA then runs on the real robot.

Results

Three tasks on a Dobot Nova 2 bimanual platform: moving a keyring between pegs, cutting sand into a bowl, and lifting a plate onto the top level of a rack.

  • Simulated cross-tool retargeting: 86.1% mean success, against 37.8% when GPT-6 Astra predicts affordances directly and at most 26.9% for prior tool-transfer methods (TAX-Pose, MAGIC, MimicFunc, IW, R-NDF).
  • Real robot, unseen tools (50 episodes per task): 35.3% mean success, against 4.7% for π0.5, 25.0% for the agent controlling the robot directly and 47.3% with human-provided affordances. On the source tools: 46.0% against 22.0% for π0.5.
  • Without simulation refinement, unseen-tool success drops to 22.0%. Going from one to five source tools raises simulated keyring-task success on unseen tools from 56.2% to 81.3%.

Limitations

The method covers only tool transfers that retargeting a demonstrated trajectory can solve. The VLA can also fail to follow correct affordance guidance; even with human-provided affordances, unseen-tool success is 47.3%.

Figures

From Figure 1: From demonstrations with a few source tools, A2-VLA reconstructs the scenes in simulation, extracts an...
From Figure 1: From demonstrations with a few source tools, A2-VLA reconstructs the scenes in simulation, extracts and validates functional-region features, and uses them to train a VLA. For an unseen tool it extracts the same features, checks them in simulation and passes them to the VLA.
From Figure 2: (a) The agent reconstructs demonstrated scenes, picks task-relevant affordance features, writes extrac...
From Figure 2: (a) The agent reconstructs demonstrated scenes, picks task-relevant affordance features, writes extraction skills and tests cross-tool retargeting in simulation, revising them from diagnostic feedback. (b) Encoded features join image, language and robot-state tokens to condition the π0.5 action expert, which outputs robot actions and an auxiliary affordance trajectory.
From Figure 6: Real-robot keyframes with different tools. In Pnp Ring a hook region near the tool tip lifts the ring;...
From Figure 6: Real-robot keyframes with different tools. In Pnp Ring a hook region near the tool tip lifts the ring; in Cut Sand a working edge cuts the sand and moves it into the bowl; in Reset Plate part of the tool forms a support plane that lifts the plate onto the upper level.
Abstract

Tools with different shapes can have similar functions, but transferring manipulation skills across tools requires understanding how tool geometry relates to the specific task and adjusting the manipulation strategy accordingly. The key challenge is to discover the common affordances that contribute to task completion across different tools and verify whether they support new tools in performing similar work. We present A$^2$-VLA, a tool-adaptation framework that combines an embodied agent with a VLA policy. It treats tool affordances as hypotheses the agent can construct, test, and revise. Given teleoperated demonstrations with a few source tools, the agent reconstructs the scenes in simulation and autonomously determines task-relevant affordance features: functional regions and their key geometric properties. It develops affordance-extraction skills and refines both the features and skills through autonomous experiments in the reconstructed scenes. Validated extraction skills accumulate in a reusable skill library for unseen tools. Finally, A$^2$-VLA uses encoded functional-region features as conditioning inputs to a $π_{0.5}$ backbone during training and inference. Evaluation examines affordance reliability in reconstructed scenes and real-robot task success on unseen tools. Our method achieves 86.1% success in simulated cross-tool trajectory retargeting based on affordances. On three real-robot tool-use tasks, A$^2$-VLA improves task success by 30.7 percentage points over $π_{0.5}$.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.