$Z$-sight: Unifying Vision-Language-Action and Latent World Modeling
TL;DR Z-sight gives a VLA a foresight expert that predicts future frames as tokens in the VLM's own visual feature space and denoises them together with the actions. On six real Unitree G1 loco-manipulation tasks it succeeds 73.3% of the time, 20 points above the best baseline, at 110 ms per action chunk.
What they do
World action models predict future video alongside actions but carry the cost of a video generator. Latent alternatives either align internal features to future embeddings or predict the features of a frozen external encoder. Z-sight instead predicts the future as spatial tokens of the VLM’s own vision encoder, and its action expert reads that prediction while both are denoised together. It is trained on human video and robot data and tested on a real humanoid and three simulation benchmarks.
How it works
- Three streams. A Qwen3.5-2B VLM is coupled with a foresight expert and an action expert in a mixture-of-transformers (3.18B parameters in total). The foresight expert predicts future visual tokens by flow matching; the action expert attends to them.
- Co-evolving targets. Future targets come from a stop-gradient EMA copy of the policy’s own causal video encoder, and the future-prediction loss never reaches the VLM, which guards against collapse and forgetting.
- Joint denoising. At inference both streams share 10 Euler steps with no pixel decoding.
- Training. VLM pre-training on about 2,000 h of egocentric human video (EgoDex, EgoVerse), mid-training on about 2,750 h of AgiBot World and Humanoid Everyday, then post-training on the target robot. On the G1, actions drive the SONIC whole-body controller.
Results
- Real Unitree G1, six tasks, 10 trials each: 73.3% success (44/60) vs. 53.3% for Fast-WAM, 40.0% for Ψ0, 36.7% for GR00T N1.7 and 26.7% for π0.5. On the four multi-stage tasks: 72.5% vs. 50.0% for Fast-WAM.
- Simulation: 81.9% on the SIMPLE humanoid benchmark (+9.6 points over GR00T N1.7), 97.7% on LIBERO, and 82.44% on clean RoboTwin 2.0.
- Removing the future-prediction loss drops SIMPLE Hard from 73.1% to 25.0% and the real G1 Dog task from 90% to 30%. Feeding in another episode’s prediction cuts SIMPLE success to 11.7%.
- 109.5 ms per 30-step chunk on an RTX 4090, about 45 ms of it for the foresight stream.
Limitations
The foresight stream predicts the future from the observation alone, without conditioning on candidate actions, so it cannot compare the outcomes of different actions or plan at test time. On randomized RoboTwin 2.0 scenes it reaches 39.00%, below π0.5’s 46.00%.
Videos
Figures



Abstract
We introduce $Z$-sight, a unified robot foundation model that couples vision-language-action (VLA) and generative latent world modeling. Video-based world action models (WAMs) show the value of joint future-and-action generation for control, yet inherit the heavy cost of video generators. Recent latent-space policies avoid pixel generation, yet typically align intermediate features or rely on frozen target encoders, decoupling future prediction from the VLM's native feature space. $Z$-sight instead predicts the future directly in the visual token space in which its VLM perceives the present. Coupling a foresight expert and a continuous action expert with the VLM in a mixture-of-transformers, $Z$-sight jointly denoises future spatial tokens and actions at inference without pixel reconstruction or external encoders. Trained on human video and robot trajectories, $Z$-sight achieves 81.9% success on SIMPLE (+9.6 points over the strongest baseline) and 73.3% across six Unitree G1 tasks (+20.0 points) at 110 ms per chunk, while remaining competitive on tabletop benchmarks.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
