ELA: End-to-End Latent Action Policy Learning from Video Demonstrations

Cheng-Chun Hsu, Peter Stone

November 2026

Policy rollouts on the five real-world tasks, from stacking bowls to opening an oven.

TL;DR ELA trains a latent action model, a latent-action policy and a small action decoder together, so an imitation policy can learn from human or unlabeled robot video plus a few action-labeled demos. With 10 robot demos and 40 human videos per task, it raises real-robot success from 48% (Diffusion Policy) to 70%.

What they do

Latent-action methods let a policy learn from video without action labels, but they usually learn the latent space first and train the policy on it afterwards. The resulting latents can be hard to predict from images and hard to decode into robot commands. ELA trains the latent action model, a latent-action policy and an action decoder together, so the latent space is shaped for both.

How it works

Latent actions. A transformer inverse dynamics model with register tokens and iFSQ quantization maps a pair of frames to a latent action; a forward dynamics model has to reconstruct the later frame from it.
Policy and decoder. An unchanged behavior-cloning policy (Diffusion Policy or UWM) predicts the latent action from the current image. A 4M-parameter transformer decoder, trained only on action-labeled data, turns it into a chunk of robot actions. Gripper commands come from a separate head.
Joint training without collapse. Sending policy gradients straight into the latent model collapses the latents, so both models are aligned to frozen DINOv3 features in alternating passes that freeze one side and update the other.

Results

LIBERO-10 with 5 of 50 demos labeled: ELA lifts Diffusion Policy from 31% to 54% and UWM from 19% to 59%. With 5 or 10 labels both beat ATM and AMPLIFY, which need external point trackers.
Real Franka, five everyday tasks, 10 robot demos and 40 human videos each, 20 trials per task: 70% average success, versus 48% for DP, 38% for Dynamo and 8-33% for LAPA, Villa-X, UniVLA and LAPA-DINO latents.
Removing the representation alignment drops success from 0.83 to 0.45 on a subset of tasks.

Limitations

Not yet tested on large cross-embodiment datasets; RGB only, with no depth or 3D priors; no highly dexterous or precision tasks. On open-oven, ELA (30%) did not beat plain DP (35%), which the authors put down to precise articulated motion being hard to learn from human video.

Videos

Robot demonstrations side by side with human video demonstrations of the same tasks.

Figures

From Figure 1: Standard behavior cloning cannot use action-free video. ELA puts latent actions and a lightweight acti... — From Figure 1: Standard behavior cloning cannot use action-free video. ELA puts latent actions and a lightweight action decoder between the policy and the robot, so the policy trains on both action-labeled demonstrations and video.

From Figure 2: In training, an inverse dynamics model infers latent actions from image pairs; they supervise a forwar... — From Figure 2: In training, an inverse dynamics model infers latent actions from image pairs; they supervise a forward dynamics model that reconstructs the future frame and an action decoder that predicts robot actions, and a latent action policy learns to output them. At deployment the policy predicts latent actions from the current image and the decoder turns them into robot actions.

From Figure 3: Joint training alternates two passes. In the LAM-update pass the policy is frozen and the alignment, a... — From Figure 3: Joint training alternates two passes. In the LAM-update pass the policy is frozen and the alignment, action and forward-dynamics losses update the latent action model; in the Policy-update pass the latent action model is frozen and the alignment and policy losses update the policy.

From Figure 4: LIBERO-10 success with N = 5, 10 or 20 of the 50 demonstrations per task action-labeled. The dashed li... — From Figure 4: LIBERO-10 success with N = 5, 10 or 20 of the 50 demonstrations per task action-labeled. The dashed lines mark DP and UWM trained on all 50 labeled demonstrations.

Abstract

Robot imitation learning is limited by the scarcity of action-labeled data, while action-free demonstrations such as videos are abundant. Recent latent-action methods learn action-relevant representations from observation transitions, but they do not optimize latent actions for downstream control, leading to latent representations that are difficult for policies to model and difficult to accurately decode into continuous robot actions. This paper introduces ELA, an end-to-end framework for learning from video demonstrations via latent actions jointly optimized for policy learning and action decoding. ELA jointly trains a latent action model, latent action policy, and action decoder, enabling policies to leverage abundant video data with only a handful of action-labeled robot demonstrations. Experiments on simulated and real-world manipulation tasks show that ELA consistently outperforms prior latent action baselines. Moreover, it integrates seamlessly with existing imitation learning methods through lightweight 4M-parameter action decoders, requiring no changes to policy architectures or training objectives. Additional materials can be found on: https://end2end-latent-action.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong