SWAM: Scaling World Action Models with Cross-Modal Alignment

Yu Zhou, Xinyuan Xia, Xuankun Yang, Yinghao Yang, Wendi Chen, Bowen Liu, Jianhan Ma, Weicheng Duan, Jun Lv, Cewu Lu, Yanwei Li

November 2026

TL;DR A study of how to pretrain world action models on about 47,000 hours of general video, egocentric video, UMI and robot demonstrations while keeping the video model's language-video alignment. The resulting recipe reaches 61.56% on RoboTwin under randomized scenes and 40.00% on three real bimanual tasks, against 23.33% for the strongest baseline.

What they do

A world action model (WAM) starts from a pretrained video generator and adds action prediction. SWAM asks which data and supervision keep the video model’s language-video alignment intact while actions are learned. The authors curate about 47,000 hours of general video, egocentric video, handheld-gripper (UMI) demonstrations and robot demonstrations, and run controlled ablations on RoboTwin 2.0 to derive a pretraining recipe.

How it works

Model. A Mixture-of-Transformers with a video branch initialized from Wan2.2-TI2V-5B and a randomly initialized action branch. Attention flows one way, from video to action, so action-free video can still train the video branch. Both branches use flow matching; each chunk has 2 video latent frames and 48 action steps.
Language only through video. The action branch has no text cross-attention and reads the instruction through video features.
Data. About 11k hours of general video, 11k of egocentric video, 15k of in-house UMI data and 10k of bimanual robot data, filtered by a VLM and by action-validity checks. Actions use a shared 16-dimensional bimanual end-effector representation.
Captions. A VLM splits videos into subtasks and writes brief, medium and detailed descriptions plus scene and lighting notes.

Results

Language: on RoboTwin Hard (randomized scenes), detailed subtask captions with environment information reach 55.04%, against 47.20% for brief ones.
Data mix: equal egocentric-to-UMI sampling gives 53.84% Hard, against 42.56% for egocentric only and 51.28% for UMI only. Routing language through video gives 57.36%, against 52.48% for a standard dual-stream model.
Final model on RoboTwin 2.0: 88.10% Easy and 61.56% Hard; OpenWAM gets 89.30% and 48.70%.
Real bimanual Flexiv Rizon 4, three tasks, 30 trials each: 40.00% Hard success against 23.33% for OpenWAM. Without multisource pretraining the model completes no real task.

Limitations

The authors note limited support for complex reasoning and explicit multi-stage planning, and a limited history that may hinder long-range tasks. Ablations use 25 evaluation episodes per task, and each real task is post-trained on 15 to 26 hours of task-specific data.

Figures

From Figure 2: Multisource data and model overview. (a) Composition of the 47k-hour pretraining data pool. (b) The wo... — From Figure 2: Multisource data and model overview. (a) Composition of the 47k-hour pretraining data pool. (b) The world action model architecture, with video and action branches connected through shared masked attention.

From Figure 4: Multisource training data and curation pipeline. The upper panel shows representative examples from fo... — From Figure 4: Multisource training data and curation pipeline. The upper panel shows representative examples from four data sources. The lower panel illustrates visual checks, action checks, format conversion, subtask segmentation, and caption generation. Captions include action descriptions at three levels of detail, along with scene and lighting information.

From Figure 5: Scaling pretraining improves control generalization, plotted against training video hours. (a) RoboTwi... — From Figure 5: Scaling pretraining improves control generalization, plotted against training video hours. (a) RoboTwin SR under the Easy (C2C) and Hard (C2R) settings. (b) Real-world SR and PSR under the Hard setting, averaged across three tasks. The 0-hour setting uses the video-model initialization without additional pretraining and achieves no success in real-world evaluation.

From Figure 8: Generalization examples from head and external camera views. Fruit placement and rainbow stack demonst... — From Figure 8: Generalization examples from head and external camera views. Fruit placement and rainbow stack demonstrate generalization to new objects, containers, and scene appearances.

Abstract

World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong