From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models

Guoqiang Liang, Yiqi Lin, Yanzhe Chen, Kevin Yuchen Ma, Mike Zheng Shou

November 2026

TL;DR MetaWAM packs proprioception, actions and value into a single DCT-encoded latent frame of a pretrained video diffusion model instead of one frame per signal. It reaches 68.9% on RoboCasa with 39% lower inference latency and 85.0% on three real Franka tasks.

What they do

World Action Models adapt a pretrained video diffusion model to predict robot actions and future observations together. Low-dimensional signals (proprioception, action chunks, value) must enter a latent space shaped by natural video; prior work spends a full latent frame on each or adds dedicated encoders and decoders. MetaWAM writes all of them into one “meta-frame” in the frequency domain, which a frozen denoiser treats as closer to its prior.

How it works

Diagnostic. The relative reconstruction error of a frozen Cosmos-Predict2 denoiser measures how far an injected latent sits from the video prior. DCT-encoded signals are corrected less than spatially tiled or VAE-encoded ones.
Meta-frame. Each signal gets a disjoint channel band; its values go into fixed DCT coefficient slots and a per-channel 2D inverse DCT lifts them to a spatial latent. Readout is a DCT plus slot lookup, with no learned decoder.
Sequence. One input meta-frame (task type, current proprioception), six visual frames from three cameras, and one output meta-frame (actions, future proprioception, value): 8 frames instead of 11.
Slot-aware routing. Band-level masks mark each signal as clean conditioning or denoising target, so one model trains as policy, world model and value function.

Results

RoboCasa, 24 tasks with 50 demos each: 68.9% average success against 67.1% for Cosmos Policy; inference takes 0.291 s per query instead of 0.479 s.
LIBERO: 98.1% average (Cosmos Policy 98.5%). LIBERO-Plus: 83.4% overall against 82.2%.
Real Franka Panda (Sweep, Place Cup, Build Block; 80 demos and 20 trials per task): 85.0% in distribution and 68.3% out of distribution, against 78.3% and 63.3% for Cosmos Policy and 71.7% and 46.7% for pi0.5.
Among the four packings, the unified meta-frame also gives the best future-view PSNR, future-proprioception error and value error.

Limitations

The fixed band layout and loss weights may need adapting when signal scales or task importance change. Instruction following is limited by the Cosmos-Predict2 backbone. On LIBERO-Plus, background (72.7%) and robot initial-state (59.9%) perturbations score below Cosmos Policy.

Figures

From Figure 1: MetaWAM packs robot signals into one meta-frame. (a) Disjoint DCT bands route signals as inputs or den... — From Figure 1: MetaWAM packs robot signals into one meta-frame. (a) Disjoint DCT bands route signals as inputs or denoising targets for policy, world-model and value prediction. (b) Relative reconstruction error of the frozen denoiser at sigma = 1.0 (lower is better): Spatial Duplication and VAE Encoding require larger corrections, DCT Encoding is corrected least.

From Figure 2: Robot-signal injection schemes used in the diagnostic study. (a-c) Single-signal encodings: Spatial Du... — From Figure 2: Robot-signal injection schemes used in the diagnostic study. (a-c) Single-signal encodings: Spatial Duplication, VAE Encoding and DCT Encoding. (d-g) Multi-signal packings: Separate Spatial Duplication, Meta-Quadrants, Meta-Compact and the Unified Meta-Frame, which assigns signals to disjoint channel bands and lifts them to one spatial latent by an inverse DCT.

From Figure 3: Overview of MetaWAM. Normalized signals are packed into fixed coefficient slots and lifted by 2D inver... — From Figure 3: Overview of MetaWAM. Normalized signals are packed into fixed coefficient slots and lifted by 2D inverse DCT. Slot-aware masks select clean inputs and prediction targets for policy, world-model, and value learning.

From Figure 4: Real-world rollouts. For each of the three tasks (Sweep, Build Block, Place Cup): a task reference, a ... — From Figure 4: Real-world rollouts. For each of the three tasks (Sweep, Build Block, Place Cup): a task reference, a generated trajectory and a real rollout.

Abstract

Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong