STVLA: Spatial-aware and Temporal-memory for Vision-Language-Action Models

Zhuo Han, Yeqiang Wang, Xinran Wang, Hongwei Han, Yi Xiong, Zunian Wan, Jianbo Mao, Huan Peng, Yongcheng Liu, Wei Shan, Maoqing Yao

November 2026

Block stacking on the AGIBOT G2, placing a blue block on a raised red block.

TL;DR STVLA adds a memory of past 3D scene-geometry features, taken from a frozen visual geometry model, to the pi0.5 VLA. On three real dual-arm tasks it averages 76.7% success against 35.6% for pi0.5, including 60/60 block stacks.

What they do

Many vision-language-action policies act on the current frame alone. They lose track of which steps are already done, and 2D features say little about 3D structure such as the height of a surface. Memories built on appearance also fail once the gripper occludes the scene. STVLA extends pi0.5 with a bounded memory that stores scene-geometry features next to the usual visual tokens, so the policy can recall both what the scene looked like and how it was shaped.

How it works

Spatial tokens. A frozen 1B-parameter VGGT-Omega reads the last four head-camera frames. A trainable compressor (learned-query cross-attention and a variational bottleneck) turns the current frame’s features into 256 spatial tokens, which the action expert sees in their own prefix segment.
Memory. Paired visual and spatial tokens from up to 16 earlier steps of the episode. The current tokens retrieve from it through two cross-attention layers, and a learned gate fuses the result.
Retrieval score. A parameter-free score, similarity to the current step times normalized difference from it, biases attention toward entries that are relevant but not redundant.

Results

Real AGIBOT G2 dual-arm robot: block stacking 100% (60 trials) vs. 41.7% for pi0.5, 58.3% for Spatial Forcing and 88.3% for MemoryVLA; object organization 85% vs. 55%, 40% and 35%; phone packaging 45% vs. 10%, 10% and 15%.
Simulation averages: LIBERO 97.8% (pi0.5 96.6%), LIBERO-Mem 41.0% (38.0%), MIKASA-Robo 45.2% (32.4%; MemoryVLA 41.2%), SimplerEnv Bridge 74.0% (MemoryVLA 71.9%).
Block-stacking ablations: 95.0% without memory, 96.7% without the spatial branch, 80.0% without the retrieval bias.

Limitations

The authors note that the frozen 3D model does not always give good spatial tokens and that the short memory window limits gains on longer tasks. Results are sensitive to the bias scale and frame count: three of the five nonzero bias settings scored 71.7%, below no bias. Phone packaging still fails more than half the time, mostly at grasping and placement.

Videos

Object organization, a button-press, drawer and box sequence using both arms.

Phone packaging, packing a phone case and accessories into a box.

Figures

From Figure 2: Architecture of STVLA. Current visual and spatial tokens query a bounded memory bank through score-bia... — From Figure 2: Architecture of STVLA. Current visual and spatial tokens query a bounded memory bank through score-biased cross-attention (top), and the retrieved features are gated into the current representations to condition action generation (middle). The spatial pathway (bottom) compresses features from the frozen VGGT-Omega through pooling cross-attention and a variational bottleneck.

From Figure 4: Real-world tasks on the AGIBOT G2, from top to bottom block stacking, object organization and phone pa... — From Figure 4: Real-world tasks on the AGIBOT G2, from top to bottom block stacking, object organization and phone packaging. Each panel shows the language instruction (left), the workspace (center) and key frames of the manipulation sequence (right).

Abstract

Robotic manipulation often requires both spatial scene understanding and task history. However, many vision-language-action (VLA) policies rely primarily on current observations, without explicitly integrating spatial structure and task history. Appearance-based memories can be sensitive to occlusion and viewpoint changes, while spatial representations of the current scene alone do not preserve how the scene has changed during task execution. We introduce STVLA, a framework that incorporates features from a pretrained visual geometry foundation model into temporal memory alongside visual representations. This allows the policy to recall past scene structure as context for current action decisions. To select informative history, STVLA uses a mathematically grounded scoring rule that balances relevance to the current observation with complementary evidence from memory. Across evaluations on LIBERO, LIBERO-Mem, MIKASA-Robo, and SimplerEnv Bridge, STVLA achieves the highest average success rate among the compared methods on each benchmark. On real-world block stacking, object organization, and phone packaging, it improves success over $π_{0.5}$ by $58.33$, $30.00$, and $35.00$ percentage points, respectively. These results support integrating scene geometry into temporal memory to improve manipulation requiring both spatial precision and task history. Project website: \href{https://stvla.github.io/}{https://stvla.github.io/}

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong