← Paper Digest

Render the Robot, Generate the World: One Video World Model Across Embodiments

Yash Jangir, Karan Mirakhor, Renos Zabounidis, Muhammad Zubair Irshad, Sergey Zakharov, Yonatan Bisk, Homanga Bharadhwaj

November 2026

Futures predicted by Doppel for different embodiments, each from an initial image and the rendered motion.

TL;DR Doppel renders the robot along a planned joint trajectory and conditions a video world model on those renderings plus the actions, so one model covers robot arms, a humanoid and human bodies without embodiment labels. Across three robot-arm datasets it improves PSNR over action-only conditioning by 1.00 dB on held-out trajectories and 1.20 dB out of distribution.

What they do

Action-conditioned video world models usually take numerical joint vectors and must learn both how the robot moves in the image and how the scene responds. Doppel renders the robot along the proposed joint trajectory, using its known kinematics and camera pose, and conditions a pretrained video generator on these renderings plus the numerical actions. Because every body is expressed in image space, one model with no embodiment label trains on robot arms, a bimanual robot, a humanoid, and human hands and bodies.

How it works

  • Rendering. Forward kinematics from the URDF poses a per-robot Gaussian model (Dr. Robot), rendered from the observation camera; people are rendered from fitted MANO, SMPL-X and SMPL-H meshes.
  • Camera recovery. Missing extrinsics are recovered by optimizing the camera pose through the differentiable renderer against RGB, optical flow and DINOv2 features. Extrinsics for 45,345 demonstrations are released.
  • Conditioning. A 30.5M-parameter adapter encodes the renderings, lets render tokens cross-attend to zero-padded action vectors, and adds a per-token vector to the flow-timestep embedding of a Wan 2.1 14B image-to-video backbone.
  • Data. Doppel-3 trains on DROID, Bridge and RoboMind (Franka, WidowX, UR5e); Doppel-8 adds five more datasets, for about 593 h of training video.

Results

  • With the same data, adding renderings to the actions raises PSNR over action-only conditioning by 0.2-1.5 dB, and every Doppel variant beats a fine-tuned skeleton-conditioned baseline (OSCAR) on all six robot splits.
  • In the unseen Bridge kitchen, Doppel-8 raises moving-object PSNR from 21.50 to 24.85 dB and object SCS from 0.778 to 0.831 over action-only.
  • A shared Doppel-3 beats a shared action-only model’s average PSNR and SCS with 63.8% fewer training draws.
  • The five non-arm datasets improve robot predictions on four of six robot evaluation sets; Bridge OOD drops by 0.24 dB.

Limitations

Doppel needs accurate robot geometry, a URDF and camera calibration; unknown or rapidly changing viewpoints remain a problem. Evaluation covers only embodiments seen in training and one camera per trajectory, and all results are video-prediction metrics: the model is not used for planning or policy evaluation in the paper.

Videos

Ground truth, action-only baseline and Doppel-8 in an unseen kitchen and with new objects; the baseline moves the arm but leaves the object behind.
Bimanual YAM arms, from a real first frame and the rendered commanded motion to Doppel's prediction (2x speed).

Figures

From Figure 1: Doppel conditions one shared video world model on future renderings of the embodiment, for manipulator...
From Figure 1: Doppel conditions one shared video world model on future renderings of the embodiment, for manipulators, humans and a humanoid, to predict how the scene responds.
From Figure 4: Doppel-8 predictions on held-out test clips from the five added datasets: a bimanual YAM robot, a huma...
From Figure 4: Doppel-8 predictions on held-out test clips from the five added datasets: a bimanual YAM robot, a humanoid with dexterous hands, a human body, human hands and a human upper body (top to bottom). Each row shows the initial image, the future embodiment renderings used as conditioning and the predicted frames, all from one model with no embodiment label.
From Figure 5: Ground truth, action-only baseline and Doppel on trajectory and out-of-distribution generalization. Do...
From Figure 5: Ground truth, action-only baseline and Doppel on trajectory and out-of-distribution generalization. Doppel produces robot-object interactions that follow the given action trajectory more plausibly than the baseline.
From Figure 7: Training efficiency of Doppel-3 versus single-dataset action-only experts on DROID, Bridge and RoboMIN...
From Figure 7: Training efficiency of Doppel-3 versus single-dataset action-only experts on DROID, Bridge and RoboMIND. Doppel reaches higher validation and OOD PSNR and SCS at comparable training draws.
Abstract

Predicting the consequences of an agent's actions requires modeling both the body's motion and the environment's response. Most video world models attempt to learn both from numerical actions, even though geometry and kinematics already specify the intended motion. We introduce Doppel, an embodiment-agnostic video world model conditioned on camera-aligned renderings of future body configurations and numerical action trajectories. By explicitly providing the intended motion, Doppel can focus on predicting object motion and interaction-driven changes in the environment. Rendered motion also provides a common visual interface across different embodiments and action spaces. To train on robot datasets without camera extrinsics, we recover camera poses and align rendered trajectories with recorded observations. We jointly train a single shared video backbone and conditioning adapter on eight datasets spanning eight robot and human embodiments, from single- and bimanual robot arms to a dexterous humanoid and human hands and bodies, using about 600 hours of video without embodiment labels. Across three robot-arm datasets, Doppel improves PSNR over action-only baselines by 1.00 dB on held-out trajectories and 1.20 dB under out-of-distribution evaluation, while predicting more consistent object motion. Training on all eight embodiments (Doppel-8) improves robot predictions over the three-robot model (Doppel-3) on four of six evaluation sets. Project page: https://doppelwm.github.io/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.