Future Representation Distillation for Imagination-free World Action Models

Jaehyeong Jo, Sangwon Jang, Sihyun Yu, Seonghyeon Ye, Yekyung Nah, Hojung Jung, Kimin Lee, Sung Ju Hwang

November 2026

Side-by-side real-world rollouts of DreamFree, DreamZero and π0.5 on a DROID Franka, 1x speed.

TL;DR DreamFree turns a pretrained video diffusion model into a visual encoder that predicts future-frame features from the current image, giving a robot policy video-model foresight without generating any video. On 10 real DROID tasks it succeeds 64% of the time (DreamZero 56%, π0.5 16%) while running at 11 Hz, 17.8x faster than DreamZero.

What they do

World Action Models (WAMs) get robot control out of pretrained video diffusion models, but usually by generating future video at test time, which is slow and fails when the predicted video is wrong. DreamFree keeps the video model’s foresight but skips generation: a visual encoder learns to predict, from the current frame alone, the features a frozen video model computes over the current and future frames. An action module then reads actions off those features.

How it works

Predictive visual encoder. Initialized from a Wan2.2 video backbone (5B or 14B). An attention mask keeps current-frame tokens from attending to future tokens, so their features depend only on the current observation and instruction.
Future representation distillation. A small projector maps current-frame features onto the frozen teacher’s representation of the whole noised clip, trained with a cosine loss plus the original flow-matching video loss. No action labels are needed, so any demonstration video can be used.
Action module. A Perceiver-style readout compresses encoder tokens into 64 tokens; a 1B DiT predicts 32-step action chunks with flow matching. At test time the encoder runs once, with no video denoising.

Results

Real DROID Franka, 10 tasks with new objects or arrangements, 5 rollouts each: 64% success and 83% task progress, vs 56% / 71.8% for DreamZero and 16% / 33.5% for π0.5. DreamZero fails every trial of towel covering, hat hanging and bolts in bowl.
11 Hz closed-loop control with the 14B backbone: 17.8x faster than DreamZero, 1.1x the latency of π0.5.
RoboCasa-Kitchen (24 tasks): 71.7% with the 14B backbone, 13.2 pp above 5B, and 74.4% with extra action-free videos. LIBERO average 98.6%.
Learning new tasks from videos without actions: composite RoboCasa tasks rise from 0% to 21.7%; videos from a dual-arm Franka lift ALOHA-AgileX success from 6.7% to 30%.

Limitations

Pump pressing succeeded once in 5 trials despite contact every time. Video-only fine-tuning cannot add a motor skill the action module lacks. The pipeline has three training stages, DROID training took about a week on 16 H200s, and extra video data was not tested on DROID. The bimanual YAM results are qualitative only.

Videos

Bimanual YAM arms loading plates into a tabletop dish rack.

Bimanual YAM arms picking up plastic bottles next to a bin.

Figures

From Figure 1: DreamFree distills predictive knowledge from a frozen video diffusion model into a visual encoder, so ... — From Figure 1: DreamFree distills predictive knowledge from a frozen video diffusion model into a visual encoder, so actions are predicted from the current observation without generating future videos.

From Figure 2: Two training stages. (a) A pretrained video diffusion backbone becomes the visual encoder, trained wit... — From Figure 2: Two training stages. (a) A pretrained video diffusion backbone becomes the visual encoder, trained with its video generative loss under a structured attention mask and by distilling predictive representations from the frozen video model. (b) An action readout module and action decoder learn to predict actions from the encoder features.

From Figure 7: Mean inference latency per action chunk on an H200 (left), where DreamFree is 17.8x faster than DreamZ... — From Figure 7: Mean inference latency per action chunk on an H200 (left), where DreamFree is 17.8x faster than DreamZero, and task completion time for putting 2, 4 or 6 blocks into a bowl (right).

Abstract

World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong