A$^2$-VLA: Adapting Vision-Language-Action Policies to New Tools through Agentic Affordance Discovery

TL;DR An LLM agent rebuilds tool-use demonstrations in simulation, works out which part of a tool does the job (a hook, an edge, a support surface), and feeds those features to a π0.5 policy. On a real robot with unseen tools, success rises from 4.7% for plain π0.5 to 35.3%.
What they do
A VLA policy trained on demonstrations with a few tools tends to fail when handed a differently shaped one. In A2-VLA an LLM agent (GPT-6 Astra) works out which part of each tool does the job, such as a hook, a cutting edge or a support surface, tests that hypothesis in simulation, and passes the resulting geometric features to a π0.5 policy. A new tool then needs only an RGB-D inspection, not new demonstrations.
How it works
- Real-to-sim. The agent rebuilds each teleoperated demonstration (scene, tools, recorded robot motion) in simulation.
- Affordance discovery. Comparing source tools, it picks a functional region and its geometry (position, tangent, opening or cutting direction) and writes code to extract them. It tests these features by retargeting a demonstration to other tools in simulation and revises regions, features or code from the outcome. Validated extraction skills go into a reusable library.
- Affordance-conditioned VLA. The 12-D feature vector becomes one extra token for π0.5, an auxiliary head predicts the future affordance trajectory, and 10% of the training data are counterfactual retargeted trajectories so the policy learns to follow the features.
- New tools. The agent reconstructs the unseen tool, extracts and refines its features in simulation, and the VLA then runs on the real robot.
Results
Three tasks on a Dobot Nova 2 bimanual platform: moving a keyring between pegs, cutting sand into a bowl, and lifting a plate onto the top level of a rack.
- Simulated cross-tool retargeting: 86.1% mean success, against 37.8% when GPT-6 Astra predicts affordances directly and at most 26.9% for prior tool-transfer methods (TAX-Pose, MAGIC, MimicFunc, IW, R-NDF).
- Real robot, unseen tools (50 episodes per task): 35.3% mean success, against 4.7% for π0.5, 25.0% for the agent controlling the robot directly and 47.3% with human-provided affordances. On the source tools: 46.0% against 22.0% for π0.5.
- Without simulation refinement, unseen-tool success drops to 22.0%. Going from one to five source tools raises simulated keyring-task success on unseen tools from 56.2% to 81.3%.
Limitations
The method covers only tool transfers that retargeting a demonstrated trajectory can solve. The VLA can also fail to follow correct affordance guidance; even with human-provided affordances, unseen-tool success is 47.3%.
Figures



Abstract
Tools with different shapes can have similar functions, but transferring manipulation skills across tools requires understanding how tool geometry relates to the specific task and adjusting the manipulation strategy accordingly. The key challenge is to discover the common affordances that contribute to task completion across different tools and verify whether they support new tools in performing similar work. We present A$^2$-VLA, a tool-adaptation framework that combines an embodied agent with a VLA policy. It treats tool affordances as hypotheses the agent can construct, test, and revise. Given teleoperated demonstrations with a few source tools, the agent reconstructs the scenes in simulation and autonomously determines task-relevant affordance features: functional regions and their key geometric properties. It develops affordance-extraction skills and refines both the features and skills through autonomous experiments in the reconstructed scenes. Validated extraction skills accumulate in a reusable skill library for unseen tools. Finally, A$^2$-VLA uses encoded functional-region features as conditioning inputs to a $π_{0.5}$ backbone during training and inference. Evaluation examines affordance reliability in reconstructed scenes and real-robot task success on unseen tools. Our method achieves 86.1% success in simulated cross-tool trajectory retargeting based on affordances. On three real-robot tool-use tasks, A$^2$-VLA improves task success by 30.7 percentage points over $π_{0.5}$.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
