DreamFAR: Real-Time and Long-Horizon Interaction with Diverse Objects

TL;DR A video world model trained from scratch on bimanual teleoperation data, which remembers the frames where the robot makes or breaks contact and streams at 10 Hz. Over 60-s rollouts on 1,000 held-out episodes it averages 16.4 dB PSNR, against 15.8 for Cosmos-Predict 2.5 and 14.5 for a size-matched WEAVER.
What they do
DreamFAR is an action-conditioned video world model for a bimanual robot, trained from scratch on teleoperation data with no internet-video pretraining. It streams frames at 10 Hz, so a policy or a human on leader arms can act inside it in closed loop, and keeps contact-rich rollouts coherent for a minute or more.
How it works
- Tokenizer. A fine-tuned SSDD encoder maps each of three 256x256 camera views to a 16x16 grid of latents projected onto the unit sphere, which limits scale drift.
- Contact-based memory. An auxiliary head predicts one contact bit per arm; when one flips (a grasp or a release), that frame enters a keyframe bank. Up to eight keyframes, retrieved by end-effector-pose similarity, join a 5-s window of recent frames.
- Streaming transformer. 1.9B parameters with a learned, never-evicted sink frame, trained with per-frame noise levels (diffusion forcing). At inference 16 frames at staggered noise levels are denoised together over a rolling KV cache, so cost per step stays constant.
- Data. An internal teleoperation dataset on YAM arms; 256 H200 GPUs for six days.
Results
- Against baselines retrained on the public ABC dataset (DreamFAR and WEAVER both at about 0.8B), 60-s rollouts on 1,000 held-out episodes average 16.4 dB PSNR, versus 15.8 for Cosmos-Predict 2.5, 14.5 for WEAVER and 11.0 for IWS. FID rises from 15 to 32 over the minute, versus 30 to 73 for Cosmos-Predict 2.5.
- Policy evaluation with two π0 checkpoints on three tasks (32 episodes each): mean task progress inside the model and on the real robot correlates at Pearson r = 0.71.
- Training π0 on 75 real plus 75 demonstrations collected inside DreamFAR nearly doubles real-only task progress, to 63% over 10 real trials.
- Contact memory beats fixed-stride and no memory on wrist-view PSNR (19.25 vs 18.99 vs 18.66).
Limitations
One YAM station, one embodiment and parallel grippers only. The model is optimistic: it reports higher progress than the real robot and preserves the policy ranking on two of three tasks. It is trained on mostly successful demonstrations, so the authors warn it should not be the only check before deploying a policy.
Figures




Abstract
World models have shown great potential in serving as a simulator for robot manipulation. However, existing models are either fit to a single environment or lack ability to simulate long-horizon contact-rich events. We present DreamFAR, a world model whose dynamics model is trained from scratch on a large robot-native teleoperation dataset, with no internet-video pretraining. We present an architecture and learning recipe designed for training on robot-native data. DreamFAR is significantly smaller than the high-performance video generation models, yet shows better understanding of contact and causality, and runs 10 FPS on a commercial GPU. Experimentally, DreamFAR significantly outperforms existing models in predicting the long-horizon future. We further demonstrate various applications where DreamFAR can be used, including policy evaluation and mixed-data co-training. Project page: https://www.yimingdou.com/DreamFAR/
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
