Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation

Weiduo Yuan, Ze Yuan, Sihan Liu, Jionghao Wang, Liang Heng, Zhenyu Zhao, Yue Wang

November 2026

Simulated bimanual rollout moving a red and a green block, with UMI grippers on both arms.

TL;DR Spatial-UMI-Dream pretrains a 5B world action model on 605 hours of UMI handheld-gripper data to predict future RGB, stereo-derived metric depth and actions together. It reaches 85.0% average success on six real bimanual tasks, 24.2 points above the best baseline (π0.5).

What they do

UMI handheld grippers collect demonstrations without a robot, but their moving, close-range fisheye cameras leave scene scale and distances implicit in RGB. Spatial-UMI-Dream is a 5B World Action Model (WAM), built on the Wan2.2-TI2V-5B video model, that jointly generates future RGB, metric depth and bimanual actions. The authors also build RoboTwin-UMI, a 20-task simulation benchmark with UMI fisheye views that tests transfer across lighting and robot embodiments without fine-tuning.

How it works

Metric depth. Each GenRobot DAS UMI gripper has three cameras. Two form a stereo pair; Fast FoundationStereo estimates disparity, and the depth is mapped back into the native fisheye view.
RGB-D canvas. Depth is Turbo-colormapped so the pretrained video VAE can encode it, and both grippers’ RGB and depth panels are tiled into one 320×320 frame, with no depth-specific branch.
Two-stage training on 605 h. World-only adaptation first predicts future RGB-D. Action and state encoders are then added, and the transformer jointly denoises video and action chunks with flow matching.
Deployment. Demonstrations the Dexmate Vega1 robot cannot execute are filtered out by IK replay in ManiSkill, and gripper commands are compensated for delay with a first-order response model.

Results

Real world (Vega1, six tasks, 20 episodes each): 85.0% average success against 60.8% for π0.5; on Bimanual Handover 80.0% against 45.0%.
RoboTwin-UMI in domain: 79.10% against 74.45% for π0.5 and 61.20% for Fast-WAM. After transfer to the ARX X5 arm the gap over π0.5 widens to 14.00 points; under combined embodiment and lighting shift it scores 72.15% against 61.85%.
Ablations: an RGB-only variant with matched token count loses 5.75 points in domain, and real-world success drops from 85.0% to 80.8%. World-only adaptation raises simulated success from 39.50% (Wan2.2 without UMI pretraining) to 61.25%, joint pretraining to 79.10%.

Limitations

The paper does not discuss limitations. Real-world tests use one robot with 20 to 40 demonstrations per task, and the simulated generalization tests vary only light color and two arm embodiments that share the same gripper. Each pretraining stage ran on 24 GPUs.

Figures

From Figure 3: Metric depth for UMI observations. The middle and right cameras of each gripper are rectified, Fast Fo... — From Figure 3: Metric depth for UMI observations. The middle and right cameras of each gripper are rectified, Fast FoundationStereo recovers metric depth, and the depth is mapped back into the middle camera's fisheye view to align with its RGB.

From Figure 2: Architecture and two-stage training. Dual-gripper RGB and metric depth form one RGB-D canvas encoded b... — From Figure 2: Architecture and two-stage training. Dual-gripper RGB and metric depth form one RGB-D canvas encoded by the Wan VAE. Stage 1 predicts only future RGB-D; Stage 2 starts from that checkpoint, adds UMI state and noisy actions, and predicts future RGB-D and actions together.

From Figure 4: The six real-world tasks, unimanual and bimanual, shown as egocentric keyframes that are for visualiza... — From Figure 4: The six real-world tasks, unimanual and bimanual, shown as egocentric keyframes that are for visualization only, not policy inputs.

From Figure 5: Average RoboTwin-UMI success on Aloha, the training embodiment, and on ARX X5 without fine-tuning, for... — From Figure 5: Average RoboTwin-UMI success on Aloha, the training embodiment, and on ARX X5 without fine-tuning, for Spatial-UMI-Dream and four baselines.

Abstract

The Universal Manipulation Interface (UMI) enables scalable demonstration collection independently of a specific robot. However, its moving, close-range gripper cameras entangle camera and object motion without a stable external view, leaving the metric geometry needed for precise manipulation implicit in RGB. We introduce Spatial-UMI-Dream, a 5B spatial World Action Model (WAM) pretrained on 605 hours of RGB-D UMI data through two-stage training -- world-only adaptation followed by joint world-action learning -- to jointly predict future RGB, fisheye-aligned metric depth, and actions. Stereo-derived depth makes scene scale and spatial relationships explicit within a unified dual-gripper RGB-D representation. Furthermore, evaluating UMI policies requires testing whether UMI's robot-independent demonstration collection translates into policy generalization across robot embodiments, alongside robustness to environmental changes. We introduce RoboTwin-UMI, a twenty-task simulation benchmark built on RoboTwin with UMI wrist-mounted fisheye observations, enabling controlled, repeatable evaluation of UMI policies under environment and embodiment changes separately and jointly without additional fine-tuning. On RoboTwin-UMI, Spatial-UMI-Dream achieves 79.10% in-domain success and 72.15% under combined embodiment and environment shifts, exceeding the strongest evaluated baseline in each setting by 4.65 and 10.30 percentage points, respectively. It also exhibits the smallest success-rate drop under embodiment transfer among the evaluated policies. In a matched pretraining ablation, RGB-only pretraining leads to a 5.75-percentage-point decrease in in-domain success compared with RGB-D pretraining. Across six real-world tasks, Spatial-UMI-Dream achieves 85.0% average success, exceeding the strongest baseline by 24.2 percentage points. Project page: https://weiduoyuan.com/spatial-umi-dream/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong