HiLo-WAM: Decoupling Long-Horizon Task Evolution and Local Dynamics in World Action Models

Xingyu Wang, Yanjiang Guo, Zhijing Ding, Jiaxin Fang, Zeyu Lai, Jiayu Dong, Weikang Wan, Jianyu Chen, Yuxiang Gao, Mengdi Xu

November 2026

HiLo-WAM on the Fourier GR-3 during the Build House block-assembly task.

TL;DR HiLo-WAM splits a world action model's video prediction into a sparse long-horizon stream that tracks task progress and a dense short-horizon stream for robot-object contact, both guiding one action generator. It reaches 93.70% on RoboTwin 2.0 and completes 33% of long block-assembly trials on a Fourier GR-3, against 23% for π0.5.

What they do

World action models (WAMs) predict future video and robot actions together, but usually over a short window, which says little about how a long task is progressing. HiLo-WAM splits the video prediction into two streams at different time scales: a sparse one for task history and coarse future, and a dense one for local robot-object interaction. It is tested on RoboTwin 2.0, long-horizon RoboDojo tasks, and three bimanual block tasks on a Fourier GR-3 humanoid.

How it works

Dual DiT. A Video DiT initialized from Wan2.2-5B and an Action DiT exchange information through joint attention at every layer. All three streams are trained with flow matching, each at its own sampled noise level.
Two time scales. The long-horizon stream holds 8 past and 8 future keyframes spaced 4x wider than the dense local stream, which predicts short-horizon frames; actions come in 32-step chunks.
Structured attention. Local and action tokens see the history and the nearest future keyframe, but not the distant future, so control does not lean on uncertain long-range predictions.
Progressive inference. Visual streams stop after 5 of 10 denoising steps and are cached while actions finish; steps are skipped when consecutive predictions barely change.

Results

RoboTwin 2.0 (50 bimanual tasks): 93.70% average success, vs. 92.80% for AHA-WAM and 91.85% for Fast-WAM, without embodied pretraining. Removing the long-horizon stream drops it to 90.58%.
RoboDojo long-horizon tasks: progress score 17.43 vs. 9.14 for Fast-WAM.
Real GR-3 (50 h of teleoperated data, 10 trials per task): in-domain progress 6.9/10 and 33% success vs. 5.9 and 23% for π0.5. Under lighting, background and clutter changes: 6.0 and 30%, while Fast-WAM drops to 0% success.
Progressive inference cuts latency from 1114 ms to 380 ms (2.93x) with no loss in success.

Limitations

The model is bound by the video backbone’s semantic understanding; tasks that need explicit subtask decomposition, progress estimation or higher-level reasoning may stay hard. Real-world success is about one in three trials, from 10 trials per task.

Videos

HiLo-WAM on the Fourier GR-3 pulling apart the block structure in Disassemble House.

Figures

From Figure 1: Naive WAMs model dense visual dynamics over a short local horizon, while HiLo-WAM splits visual world ... — From Figure 1: Naive WAMs model dense visual dynamics over a short local horizon, while HiLo-WAM splits visual world modeling into a sparse long-horizon process for history and coarse future evolution and a dense local process for fine-grained interaction, both coupled to actions.

From Figure 2: A shared Video DiT models the sparse long-horizon stream and the dense local dynamics stream, and an A... — From Figure 2: A shared Video DiT models the sparse long-horizon stream and the dense local dynamics stream, and an Action DiT generates actions from both. At inference the visual streams stop early and are cached while the action stream finishes denoising.

From Figure 9: Real-world success rate and progress score on the Fourier GR-3 for π0.5, Fast-WAM and HiLo-WAM on Buil... — From Figure 9: Real-world success rate and progress score on the Fourier GR-3 for π0.5, Fast-WAM and HiLo-WAM on Build House, Disassemble House and Disassemble Heart, in-domain and out-of-distribution.

Abstract

World Action Models (WAMs) improve robotic manipulation by jointly modeling visual dynamics and robot actions. However, most existing WAMs model dense short-horizon visual dynamics, capturing local interactions but providing limited long-range task context. Extending prediction horizons increases uncertainty and computation while entangling task-level evolution with fine-grained local dynamics, obscuring their dependencies and distinct roles in action generation. We propose HiLo-WAM, which decouples long-horizon task evolution from local visual dynamics while jointly modeling their interaction with robot actions. A sparse long-horizon visual stream captures historical task context and future evolution, while a dense local dynamics visual stream models short-horizon robot-object interactions. The two streams operate at different temporal scales within a unified dual-DiT architecture and guide actions through structured cross-stream attention. Progressive inference further reduces unnecessary computation through stream-wise partial denoising, joint denoising, and dynamic skipping. Experiments demonstrate strong long-horizon manipulation performance: HiLo-WAM achieves a 93.70% average success rate on RoboTwin 2.0 and a 17.43 progress score on RoboDojo long-horizon tasks, with robust in-domain and OOD performance on the Fourier GR-3 robot. Ablations confirm the complementary contributions of long-horizon task evolution and local interaction dynamics, while temporal-scale analyses highlight the importance of balancing temporal coverage and resolution. Progressive inference further achieves a 2.93x speedup while improving performance.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong