← Paper Digest

DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play

Akshay L Chandra, Iman Nematollahi, Lingyue Zhao, Daria Kulikova, Angela P. Schoellig, Fabien Despinoy, Abhinav Valada

November 2026

Real-robot rollouts at 2x speed in scene and gripper views, with successes marked.

TL;DR A world model driven by latent actions inferred from pairs of frames learns from robot play and from action-free human play. A diffusion policy imitates human demonstrations in that latent space and improves with RL in imagination. With human demonstrations only, it reaches 59.6% on CALVIN vs 35.2% for the best baseline given five robot demos, and on a real Franka, fine-tuning in imagination raises success from 53% to 70%.

What they do

A person can demonstrate a skill in seconds, but plain video carries no robot actions. DiWALI conditions a world model on latent actions inferred from pairs of frames instead of on robot commands, so human and robot play train the same dynamics. A diffusion policy imitates demonstrations in this latent space and is fine-tuned with PPO inside the world model. It runs on the robot through a decoder learned from robot play.

How it works

  • Latent-action world model. A DreamerV2-style recurrent state-space model on 64x64 scene and wrist images. An inverse dynamics model turns two consecutive embeddings into a 16-D latent action; it is trained end-to-end with no action loss on robot and human play.
  • Action decoder. An MLP trained afterwards, only on robot play and on frozen world-model features. It is conditioned on the latent state and the latent action, and outputs a chunk of five end-effector commands.
  • Policy. A diffusion policy denoises the latent actions of mostly human demonstrations. Recorded actions from robot demos are never used.
  • Improvement in imagination. A success classifier trained on demonstration frames provides the reward. PPO fine-tunes the last 10 of 20 denoising steps, with a behavior-cloning regularizer and a smoothness penalty on the latent actions.

Results

  • CALVIN, 11 tasks: with 50 human and no robot demos, average success is 59.6%. Given five robot demos, Diffusion Policy reaches 35.2% and DiWA 23.6%; CLAM-X with the same data as DiWALI reaches 8.2%.
  • Decoder ablation: training it separately raises success from 9.5% to 18.7%, and also conditioning it on the latent state raises it to 61.6%. Cutting robot play to 25% of the mix still gives 58.0%, but 1% gives 7.4%.
  • After fine-tuning in imagination, success is 75.8% with 5 robot and 50 human demos. DiWA with 50 robot demos on the same robot play reaches 69.5%.
  • Real Franka (drawer, slider, LED; 20 rollouts per skill, three seeds): success is 53% before fine-tuning and 70% after, with no extra real-world interaction.

Limitations

Human play is collected in the robot’s own workspace; using in-the-wild video is left to future work. Grounding latent actions still needs robot play. The world model stays frozen during fine-tuning, which caps the improvement. Block tasks remain hard at 64x64 resolution (21.5% at best).

Videos

Action-free human play in the real workspace, filmed by the scene camera and a wrist-mounted camera.
The robot in the CALVIN simulation benchmark, scene and gripper views.
Full video (YouTube)

Figures

From Figure 1: DiWALI trains one world model on robot play and on action-free human play, using latent actions inferr...
From Figure 1: DiWALI trains one world model on robot play and on action-free human play, using latent actions inferred from consecutive frames. A diffusion policy imitates human demonstrations in this latent space, improves in imagination, and runs through a decoder trained on robot play.
From Figure 2: (a) An inverse dynamics model infers a latent action from two consecutive observations, and the world ...
From Figure 2: (a) An inverse dynamics model infers a latent action from two consecutive observations, and the world model uses it in place of the robot command; no action labels are used. (b) An action decoder learns to map latent states and actions from robot play to the executed robot commands. (c) A diffusion policy learns to output the latent actions of human demonstrations. (d) The policy is fine-tuned with PPO on imagined rollouts scored by a frozen reward classifier.
From Figure 3: Scene (top) and egocentric (bottom) views of play data in four settings: CALVIN, CALVIN-H with a rende...
From Figure 3: Scene (top) and egocentric (bottom) views of play data in four settings: CALVIN, CALVIN-H with a rendered human arm, the real robot, and a real human. The human data has no action labels.
From Figure 4: Real-world successful trials out of 20 on the drawer, slider and LED skills as fine-tuning in imaginat...
From Figure 4: Real-world successful trials out of 20 on the drawer, slider and LED skills as fine-tuning in imagination proceeds, averaged over three seeds.
Abstract

Learning robot manipulation is constrained by the availability of action-labeled robot data, for both imitation and world models. Humans can demonstrate skills in seconds, but their demonstrations lack the required robot actions. We introduce DiWALI, a framework that learns and improves diffusion policies within a world model trained on human and robot play. DiWALI infers latent actions from consecutive frames using an inverse dynamics model, enabling world model training without action labels, and grounds them in robot commands through a decoder trained separately on robot play. Skill policies are trained over latent actions from action-free human demonstrations and improved with reinforcement learning in imagination. On the CALVIN benchmark, DiWALI policies trained solely from human demonstrations substantially outperform baselines trained with robot demonstrations. Incorporating human play into the world model training also yields stronger policies after fine-tuning than using the same amount of robot play alone. Experiments on a physical robot show that skills learned primarily from human demonstrations are further improved in imagination with a world model trained on human and robot play. Together, these results demonstrate that human experience can support both manipulation skill acquisition and subsequent policy improvement, reducing reliance on action-labeled robot data. Videos, dataset and code are available at https://diwali.cs.uni-freiburg.de.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.