$A^3$-VLA: Automatic Perception-Guided Attention Alignment for Robot Manipulation

Shengzhe Zhang, Qi Zhang, Dazhong Shen, Chao Wang, Shaopeng Zhai, Hui Xiong

November 2026

TL;DR Add two training losses that keep a VLA's attention on the instructed object and on the robot, using regions found automatically from where the scene changes at contact. On five real kitchen tasks, A3-VLA built on pi0 succeeds 76% of the time, against 62% for pi0.5 and 55% for pi0.

What they do

VLAs fine-tuned only on an action loss can stop looking at the object the instruction names, and pick a look-alike instead. A3-VLA adds perception supervision during fine-tuning without any manual labels. It takes it from the demonstrations themselves: the target is the part of the image that sits still until the robot touches it, and then changes.

How it works

Region proposal. At the interaction moment (from gripper state or similar cues), patches that are stable before and changing after contact become the candidate target; patches that change throughout become the robot. SAM2 propagates these masks across the trajectory offline.
Attention allocation. Two losses set a lower bound (25%) on the last-layer attention that language tokens put on the target region and that the state token puts on the robot region.
Contrastive loss. Before contact, pooled target-patch features are pulled toward the instruction embedding and the remaining patches pushed away.
The deployed policy is unchanged, so inference costs nothing extra. It is applied to pi0, pi0.5 and OpenVLA-OFT.

Results

VLABench (five suites): average success rises from 0.47 to 0.64 for pi0, 0.16 to 0.40 for OpenVLA-OFT, and 0.41 to 0.67 for pi0.5.
RoboTwin 2.0 (four tasks): gains of 6.5 to 9.8 points, e.g. 0.42 to 0.52 for pi0.
Real Songling arm, five kitchen tasks, 20 trials each: A3-VLA (pi0) averages 0.76 success against 0.55 for pi0 and 0.62 for pi0.5. Select Bottles improves most, 0.75 against 0.25 and 0.40.
Ablation: dropping the language-to-image attention loss costs 12 points of average success; the contrastive and state losses cost 3 and 4.

Limitations

The region heuristic degrades when the target is occluded, small, hard to tell from the background, or moves with distractors. It also needs a coarse interaction moment from robot or task cues. SAM2 adds offline preprocessing time.

Figures

From Figure 1: A failure case on a VLABench poker-selection task. Attention from the language instruction to the imag... — From Figure 1: A failure case on a VLABench poker-selection task. Attention from the language instruction to the image in pi0 and OpenVLA-OFT, from shallow to deep layers, misses the named card and the robot picks the wrong one.

From Figure 2: Overview of A3-VLA. Candidate robot and target-object regions are proposed from frame differences arou... — From Figure 2: Overview of A3-VLA. Candidate robot and target-object regions are proposed from frame differences around the interaction moment and refined with SAM2 (left), then used for attention-allocation and contrastive losses during VLA fine-tuning (right).

From Figure 4: Attention maps from the language instruction to the image for pi0, pi0.5 and A3-VLA (pi0) on simulated... — From Figure 4: Attention maps from the language instruction to the image for pi0, pi0.5 and A3-VLA (pi0) on simulated and real-world tasks.

Abstract

Vision-Language-Action Models (VLAs) enable robots to perform manipulation tasks by leveraging Vision-Language Models (VLMs) to process visual observations and language instructions. However, the action prediction objective itself lacks fine-grained task-specific perceptual supervision, making it prone to relying on task-irrelevant visual shortcuts, leading attention away from target objects and key interaction areas, thereby limiting its operational performance and generalization ability. To address this, we propose Automatic Perception-Guided Attention Alignment for Robot Manipulation ($A^3$-VLA), a framework that derives task-specific visual guidance from the manipulation process itself. Specifically, $A^3$-VLA first generates heuristic candidate regions for target objects and the robot from interaction-induced visual changes, and refines them with a visual segmentation model offline. These regions serve as explicit perceptual guidance through two complementary objectives: an attention allocation loss that encourages sufficient focus on task-critical regions when language or robot state tokens query the visual input, and a contrastive learning loss that aligns the representations of candidate target regions with language instructions while separating them from task-irrelevant regions. Extensive experiments on simulated and real-world manipulation tasks demonstrate that $A^3$-VLA consistently improves multiple VLA backbones, with an average success rate gain of 6.5% - 25.2% across benchmarks.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong