W2A: Distilling World Priors into Action Models for Real-Time Robotic Control

Xiaopeng Zhang, Yongjin Mu, Yueyang Weng, Chengzhi Zhao, Yanjie Li

November 2026

The Franka picks up a banana and places it in the basket (2x speed).

TL;DR W2A distills a video diffusion model's intermediate features into a small action policy during training, then drops the video model at deployment. It reaches 95.9% on LIBERO at 55 ms per prediction, about 21x faster than mimic-video, and 65.3% success across 15 real Franka tasks versus 34.2% for OpenVLA.

What they do

World action models (WAMs) use pretrained video models to imagine the future before acting, which helps long-horizon manipulation but makes each action slow. W2A keeps the video model only as a training-time teacher: its world priors are distilled into a plain action model, and the video model is discarded at test time.

How it works

Stage I. Cosmos-Predict2, a language-conditioned video DiT, is adapted to robot videos with LoRA. Latents from its middle block at an intermediate denoising step are compressed with PCA to 482 dimensions (about 99% of the variance).
Stage II. An action DiT is trained from scratch with flow matching. A small Q-Former reads features out of its middle layer, arranged in the same space-time layout as the video latents, and a cosine loss aligns the two.
Deployment. Only the action DiT runs: it predicts 16-step action chunks, executes 8, and warm-starts the next prediction from the unexecuted remainder. It runs at 15 Hz on the robot.

Results

Simulation: 95.9% average on LIBERO and 93.6% on LIBERO-Long (pi0.5: 92.4%); average chain length 4.30 on CALVIN ABC to D.
55 ms per prediction on an A100, about 21x faster than mimic-video, with about 6 GB of GPU memory.
Real Franka, 50 teleoperated demos per task and 30 trials each: 78.3% pick-and-place, 65.0% stacking and 54.2% long-horizon, against 50.8/35.0/18.3% for OpenVLA. Towel folding, table clearing and USB insertion reach 76.7/63.3/50.0%.
On LIBERO, removing distillation costs 11.5 points and skipping PCA 7.3; a random teacher costs 26.7.

Limitations

Without the video model there is no explicit future imagination at test time, which may limit robustness under large distribution shift. A fixed PCA subspace may drop higher-order structure. On the real robot, W2A fails on unseen objects or tasks and when the initial scene departs far from the training layouts.

Videos

Picking up the blue cup and placing it on the plate (2x speed).

Stacking the green cube on top of the purple cube (2x speed).

Figures

From Figure 1: W2A compared with existing world action models. Existing WAMs keep a video branch at test time; W2A di... — From Figure 1: W2A compared with existing world action models. Existing WAMs keep a video branch at test time; W2A distills world priors from a video DiT into an action DiT during training and removes the video DiT entirely at test time.

From Figure 2: The W2A training and inference framework. Stage I adapts the video DiT to robot videos with LoRA and c... — From Figure 2: The W2A training and inference framework. Stage I adapts the video DiT to robot videos with LoRA and compresses its latents with PCA; Stage II trains the action DiT and aligns its Q-Former readout with the compressed video latents; inference uses the action DiT alone.

From Figure 4: Representative real-world tasks: pick-and-place, cube stacking, putting an object in a drawer, towel f... — From Figure 4: Representative real-world tasks: pick-and-place, cube stacking, putting an object in a drawer, towel folding, table clearing and USB insertion.

Abstract

World Action Models (WAMs) have recently shown great promise in robotic control, benefiting from the future imagination capability of large-scale pretrained video models. However, most existing WAMs require explicitly imagining future states at test time, incurring substantial computational overhead and latency that make them unsuitable for real-time robotic control. In this paper, we propose W2A, a two-stage distillation framework for real-time robotic manipulation that distills intermediate denoising representations from a video diffusion model into an action model during training, while entirely discarding the video model at test time. We hypothesize that although video latent representations are high-dimensional, a compressed low-dimensional subspace is sufficient to provide effective supervision for action generation. This makes it possible to transfer such information effectively into an action model. Extensive experiments in both simulation and real-world settings demonstrate competitive success rates while significantly improving inference efficiency. On LIBERO, W2A achieves a 95.9% average success rate. More importantly, W2A achieves about 21x faster inference than the representative WAM baseline.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong