One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

TL;DR O²LaF learns latent actions from unlabeled video by adapting one pretrained video diffusion transformer into both an inverse dynamics encoder and an action-conditioned video generator. Trained only on egocentric human videos, its latent actions reach an end-effector R² of 0.277 when probed on GR-1 robot demonstrations, versus 0.149 for DreamDojo.
What they do
Latent action models learn an action space from unlabeled video. An inverse dynamics model (IDM) infers a compact action for each frame transition, and a forward dynamics model (FDM) must predict the next frames from it. Prior methods learn the IDM from scratch. O²LaF initializes both models from the same pretrained text-to-video DiT (Wan2.2-TI2V-5B), adapts each with LoRA, and trains them end to end through a 32-D Gaussian latent action.
How it works
- IDM. One forward pass of the video DiT with learnable action queries. Each query reads only the two frames of its transition and yields one latent action.
- FDM. Sees the clean first frame plus the latent actions and denoises all future frames jointly under frame-causal attention. This removes the copy-the-previous-frame shortcut that autoregressive prediction allows.
- High-noise training. FDM training emphasizes high noise levels, so the noisy targets reveal little and the model must use the latent actions. IDM and FDM noise levels are sampled independently.
- Two-stage world model. For better video quality, the IDM is frozen and a bidirectional FDM is fine-tuned on its actions.
Results
- Trained only on EgoDex human videos, linear probes on GR-1 robot demonstrations reach end-effector R² 0.277, against 0.149 for DreamDojo.
- In cycle transfer (human to robot to human), reconstruction PSNR is 24.30 vs 22.61 for DreamDojo.
- In zero-shot third-person navigation transfer, camera translation error is 0.511 vs 0.985 for Olaf-World and rotation error 0.330 vs 0.701.
- Scaling the FDM from 1.3B to 14B consistently improves action decodability. A V-JEPA2 IDM probes slightly better but leaks source appearance into generated videos.
Limitations
No policy is trained or run on a robot: latent actions are judged by probing and video generation only. The structure of the latent space is poorly understood. Video diffusion rollouts are computationally expensive and expose no executable actions, so planning and closed-loop control are left to future work. Using the FDM to check grasps before execution is shown only qualitatively.
Figures




Abstract
Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O$^{2}$LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O$^{2}$LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O$^{2}$LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
