ForeAct: Geometry-Guided Action Learning from World Representations

Wei Lu, Jiaxi Zhang, Yunheng Wang, Taowen Wang, Yuetong Fang, Zhenhui Zhang, Shuning Zhang, Zizhao Yuan, Yixiao Feng, Zecui Zeng, Renjing Xu

November 2026

TL;DR ForeAct runs a pretrained video world model once on the current frame and feeds its features to an action head, with a few learnable tokens trained to predict next-frame depth. It reaches 96.6% on LIBERO, 63.5% on SimplerEnv-Bridge and 67 of 90 real-robot trials, at 168 ms per inference.

What they do

World action models get useful features for control by generating future video, which needs many denoising steps and carries no explicit geometry. Geometry-based policies add a separate 3D backbone and only describe the scene as it is now. ForeAct uses a pretrained video world model (Cosmos-Predict2-2B) as a one-pass feature extractor and teaches it to anticipate the next frame’s depth, so the action head gets features that are fast to compute and aware of upcoming geometry.

How it works

Single pass. The current RGB frame is encoded by the frozen video VAE and passed once through the video DiT as a clean latent, with no future-frame synthesis. The output visual tokens condition a 16-layer flow-matching action DiT that predicts an action chunk. No proprioceptive state is used.
Geometry tokens. Eight learnable tokens join the visual tokens under bidirectional self-attention. A lightweight readout decodes them into the relative depth of the next frame, supervised by MoGe-3 targets computed offline.
Training only. The depth loss is weighted 0.01 and the readout is removed at deployment, so inference needs no depth input and no geometry backbone.

Results

LIBERO: 96.6% average (98.4 Spatial, 98.8 Object, 97.2 Goal, 91.8 Long) without robot-action pretraining. SimplerEnv-Bridge: 63.5% average, against 61.9% for GR00T N1.5.
Real Franka, six tasks with 15 trials each: 67/90 (74.4%), against 58/90 for FastWAM and 52/90 for pi0.
168 ms per inference with about 2.1 B parameters on one H20 GPU, roughly one-third the latency of Cosmos Policy.
Ablations on LIBERO: 94.3% without geometry tokens, 94.7% with tokens but no depth supervision, 96.6% with both. Next-frame depth beats current-frame depth (95.2%).

Limitations

The paper has no limitations section. From its results: ForeAct trails Fast-WAM (97.6%) and Cosmos Policy (98.5%) on LIBERO and is weakest on LIBERO-Long; Stack Blocks in SimplerEnv reaches only 12.5%. Supervising depth further ahead (4 or 8 frames) or using more than eight tokens lowers success. Real-world tests use one arm and one fixed camera.

Figures

From Figure 1: Backbone paradigms for robot control. (a) Video-centric WAMs derive action features through iterative ... — From Figure 1: Backbone paradigms for robot control. (a) Video-centric WAMs derive action features through iterative denoising. (b) Geometry-centric policies use features from pretrained geometric backbones. (c) ForeAct extracts action features in a single video-backbone pass, shaped by next-frame depth supervision during training.

From Figure 2: ForeAct architecture. A single forward pass through the pretrained world model jointly processes the c... — From Figure 2: ForeAct architecture. A single forward pass through the pretrained world model jointly processes the clean visual latent, language embedding, and learnable future-geometry tokens. The output visual tokens are passed to the Action DiT, while the geometry tokens are decoded into next-frame relative depth for training supervision. The geometry readout is removed at inference.

From Figure 4: Next-frame depth visualization on (a) SimplerEnv-WidowX and (b) LIBERO, at early, middle and late stag... — From Figure 4: Next-frame depth visualization on (a) SimplerEnv-WidowX and (b) LIBERO, at early, middle and late stages of one episode. The RGB row contains the current input frames, the Depth row ForeAct’s predicted next-frame relative depth.

From Figure 5: Real-world evaluation. (a) Six tabletop manipulation tasks spanning object placement, stacking, forcef... — From Figure 5: Real-world evaluation. (a) Six tabletop manipulation tasks spanning object placement, stacking, forceful interaction, and multi-stage execution: Green-on-Yellow, Stack Bowls, Sandbag-to-Scale, Banana-to-Basket, Press-Bells-in-Order, and Cup-in-Drawer. (b) Experimental setup with a Franka Emika Panda manipulator observed by a fixed Intel RealSense D435i RGB camera. (c) Task-wise numbers of successful executions for pi0, FastWAM, and ForeAct over 15 trials per task. ForeAct achieves the best result on five of the six tasks and succeeds in 67 of 90 trials (74.4%), compared with 58 (64.4%) for FastWAM and 52 (57.8%) for pi0.

Abstract

Robotic manipulation requires representations that capture both scene geometry and its evolution during interaction. Existing approaches have largely treated these two aspects separately. World action models capture such dynamics by coupling action prediction with future video generation, but their reliance on iterative diffusion or rectified-flow inference leads to high computational cost, while their representations lack explicit spatial structure. Explicitly modeling scene geometry typically requires auxiliary modules that increase both parameter count and computational cost. Conversely, geometry-centric policies primarily recover observed spatial structure and often lack explicit supervision for anticipating future geometric changes during interaction. To address these limitations, we introduce ForeAct, which repurposes a pretrained video world model as a single-pass representation backbone for low-latency control. ForeAct directly processes the clean latent of the current observation in one forward pass, eliminating future-frame synthesis and iterative denoising while retaining the backbone's pretrained spatiotemporal priors. Rather than introducing a separate geometry backbone, ForeAct uses learnable tokens supervised by next-frame relative depth to shape the shared video representation through bidirectional self-attention. This compact interface turns the spatial knowledge already encoded in the backbone into a future-aware representation for action generation, with little parameter overhead. The lightweight geometry readout is used only during training and removed at deployment, enabling low-latency, geometry-aware control without depth input or an external geometry backbone. Without additional large-scale pretraining, ForeAct achieves 96.6% average success on LIBERO, 63.5% on SimplerEnv-Bridge, and a 74.4% success rate across 90 real-world rollouts.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong