DreamX: Video Diffusion Model is a Universal Latent Action Learner

Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang, Seungryong Kim

November 2026

TL;DR DreamX reads compact latent actions out of a pretrained video diffusion model, giving human and robot videos one shared action space. It beats prior latent action models on LARYBench (regression MSE 0.17 vs 0.38), and its latent-action policy closes drawers on a real bimanual robot at 96.7% subgoal success, 19.1x faster than a video-generating policy.

What they do

Latent action models learn actions from unlabeled video, but their latents often also encode viewpoint and scene appearance, which limits transfer between embodiments. DreamX builds both the latent-action encoder and the decoder from a pretrained video diffusion model (Cosmos-Predict2.5 2B). The decoder’s generative prior supplies appearance, which pushes the latents to carry motion. Human and robot videos then share one action space.

How it works

Encoder. Four learnable query tokens per latent frame pass through the backbone’s spatiotemporal attention, condensing 1,200 frame tokens into 4 latent-action tokens (300x).
Decoder. The same backbone, with separate LoRA modules, reconstructs future frames from the first frame and the latent actions in place of text, and decodes action chunks when robot labels exist. Pretraining uses 2.19k hours of video, 469 of them human egocentric.
Transfer. Latents from a human video plus a target first frame give video transfer; plus a target robot state they give robot actions. No paired human-robot data is used.
Policy. A bottleneck compresses each latent token to a 32-dim code. The backbone predicts these codes from the current frame and instruction, and a DiT head outputs joint commands; human videos supervise only the codes.

Results

LARYBench: average action regression MSE 0.17 vs 0.38 for the best baseline (DiLA); classification accuracy 69.67% vs 62.25%.
Zero-shot human-to-robot video transfer on 1,008 HumanAndRobot clips: FVD 57.44 vs 170.19 (AdaWorld) and 241.92 (VideoWorld2).
Real bimanual YAM drawer closing (45 demonstrations, 10 trials per configuration): 96.7% average subgoal success vs 50.0% for Mimic-Video, at 1.47 s vs 28.2 s per policy call. Learning the upper-drawer case from human videos alone still gives 100%.

Limitations

The paper has no limitations section. The real-robot policy is tested on one drawer-closing task, and open-loop action transfer (writing, one- and two-arm pushing) is shown only qualitatively. The authors note that transfer and policies may fail outside the evaluated settings.

Figures

From Figure 1: DreamX learns one latent action space from egocentric human videos and robot videos using a pretrained... — From Figure 1: DreamX learns one latent action space from egocentric human videos and robot videos using a pretrained video diffusion encoder and decoder. The same latents drive video transfer, human-to-robot action transfer and a compact latent policy.

From Figure 2: (a) Learnable queries read latent actions out of the video backbone; a decoder built from the same bac... — From Figure 2: (a) Learnable queries read latent actions out of the video backbone; a decoder built from the same backbone reconstructs future video, and actions when labels exist, from the latent actions and the first frame. (b) Encoder and decoder attention masks over video, query and action tokens.

From Figure 5: Zero-shot human-to-robot video transfer on HumanAndRobot ("grab red cube and pull to plate"). DreamX keeps the human motion and the robot appearance; AdaWorld degrades the image and VideoWorld2 shows a human hand.

From Figure 6: Human-to-robot action transfer without paired training data. Latent actions from a human demonstration... — From Figure 6: Human-to-robot action transfer without paired training data. Latent actions from a human demonstration, decoded into robot actions, make the robot write the letter N, push forward then left with one arm, and push forward with both arms.

Abstract

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong