DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight

Zhongxi Chen, Yi Yang, Xinyi Chen, Mingzhe Ye, Yaping Li, Zhihong Liu, Congsheng Xu, Haonan Wang, Zhijie Deng

November 2026

Real robot closing the lid of a box with a dexterous hand.

TL;DR Train a world model that predicts 3D hand-point motion and visual feature changes from depth and hand kinematics, then align a dexterous policy's latent queries to it so the policy runs on RGB alone. Success rises from 49.4% to 60.5% on 11 DexJoCo simulation tasks and reaches 58.3% on three real-robot tasks, against 50.0% for the best baseline.

What they do

A dexterous policy needs to anticipate how fingers and scene will move, often while fingers are hidden from the cameras. World-action models usually get this by generating future frames at inference. DexForesight instead learns the prediction in a separate world model during training and transfers it into the policy’s latent features, so the deployed policy needs no depth, no world model and no future decoding.

How it works

Spatial world model. A RynnBrain-initialized backbone takes RGB, language, robot state, offline depth (StereoAnyVideo on stereo demonstrations, fused by cross-attention at several layers) and a point cloud of fixed points on the hand’s URDF links, placed by forward kinematics.
Targets. Conditioned on the demonstrated controls, learnable world queries drive two experts: one predicts 3D displacements of the hand points, including occluded fingers; the other predicts changes in DINOv3 image features.
Query alignment. The world model is frozen. Policy queries read the VLM’s hidden layers, are trained to match the world queries and their pairwise similarities, and are appended to the action expert’s context.
Data. About 370 hours of demonstrations over 60 tasks; depth is added for a 100-hour subset.

Results

DexJoCo simulation, 11 tasks, 100 trials each: 60.5% overall, against 49.4% for its GR00T N1.7 backbone and 50.9% for Fast-WAM.
Real robot, three tasks, 20 trials each: 58.3% overall (Screw On Lid 50%, Close Box 45%, Insert Battery 80%), against 50.0% for WLA and 48.3% for GR00T N1.7.
Inference takes 230.4 ms, against 227.4 ms for the base QwenVLA policy.
Removing offline depth from the world model drops success to 49.7% in simulation and 43.3% on the real robot.

Limitations

The paper states no limitations. Real-robot evaluation covers three tasks, and the authors note that the WLA and Fast-WAM comparisons differ in architecture and training, so they are not one-variable controls. Depth was estimated only for the 100-hour subset whose categories match the evaluated tasks.

Videos

Real robot inserting a battery into its holder.

Real robot screwing a lid onto a cup.

Figures

From Figure 1: Demonstrations spanning 60 tasks and five spatial categories train an action-conditioned spatial world... — From Figure 1: Demonstrations spanning 60 tasks and five spatial categories train an action-conditioned spatial world model to predict visual dynamics and 3D hand flow. The learned spatial foresight is transferred into a low-level policy, enabling action generation without world-model execution or future decoding at deployment.

From Figure 2: Architecture and query alignment. The spatial world model learns action-conditioned hand motion and vi... — From Figure 2: Architecture and query alignment. The spatial world model learns action-conditioned hand motion and visual changes from RGB, offline scene depth, hand geometry, and language. Demonstrated controls condition the world queries through the query adapter and are also supplied to both prediction experts. Queries from the frozen world model supervise policy queries, which condition the action expert.

From Figure 3: Real-robot data and training. Recording hours and example frames for the five task categories. The fin... — From Figure 3: Real-robot data and training. Recording hours and example frames for the five task categories. The final stage uses 100 hours with depth estimated offline by StereoAnyVideo.

From Figure 4: Experimental setup and tasks. Top: the Sharpa North robot with 22-DoF Sharpa Wave hands and the three ... — From Figure 4: Experimental setup and tasks. Top: the Sharpa North robot with 22-DoF Sharpa Wave hands and the three real-robot tasks. Bottom: the Franka Panda with a 16-DoF Allegro hand and example simulation tasks.

Abstract

Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline.Project page https://dexforesight.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong