Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models

Jiachun Jin, Zetong Zhou, Haonan Wang, Xiao Yang, Jun Zhu, Zhijie Deng

November 2026

Real-robot gear assembly, then spoon scooping, at 4x speed.

TL;DR Language descriptions of atomic actions align the latent actions of unpaired human and robot videos, so egocentric human video can train the same action expert that drives the robot. On a 63-DoF robot with two dexterous hands, adding task-specific human video raises average success from 53.75% to 72.5%.

What they do

Latent action models let a policy learn from videos without action labels, but latents learned only by reconstructing each clip tend to cluster by scene rather than by action, so a human doing a task and a robot doing the same task need not get similar latents. SALA uses short language descriptions of each clip’s atomic action (“grasp the cup”) to align human latents with robot latents, without paired data. The aligned latents then let egocentric human video supervise a VLA’s action expert in both pre-training and per-task post-training.

How it works

  • Tokenizer. Each clip (two frames, encoded with frozen DINOv2) becomes 8 latent tokens through a SoftVQ quantizer with 256 codes. Human and robot clips are trained to reconstruct the future frame’s features; robot clips also reconstruct their action chunk, which ties robot latents to control.
  • Language alignment. Text-embedding similarity between a human clip’s description and each robot clip in the batch gives a soft neighborhood that the human clip’s latent similarities must match. A stop-gradient keeps the robot latents fixed as anchors. With hard VQ the alignment loss stays near its initial value.
  • Policy. A pi0-style model (Qwen3-VL-4B plus a 0.6B flow-matching action expert) whose latent queries predict the tokenizer’s tokens. Training runs in three stages: human video only, then robot teleoperation, then task teleoperation plus human demonstrations.

Results

  • Real robot (wheeled humanoid, two 22-DoF hands, 63-DoF action space; 2,678 h human video, 888 h teleoperation): average success is 72.5% on gear assembly, bimanual pouring, flower insertion and spoon scooping, above GR00T N1.7 and a vanilla VLA on every task.
  • More task-specific human video (10%, 50%, 100%) raises success from 53.75% to 72.5%. Validation action error falls with each pre-training budget (0, 473, 1,103, 2,678 h).
  • RoboTwin 2.0, with another arm’s videos standing in for human video: 80.2% clean and 48.6% randomized success; without alignment 75.5% and 36.8%.
  • Human-to-robot retrieval of the same task rises from 3.8% to 34.2% with alignment.

Limitations

The tokenizer needs atomic-action descriptions for its training clips. The pre-training scaling result is offline validation error from one run per setting, up to 2,678 h; whether it continues and how it translates into real-robot success is open.

Videos

Real-robot spoon scooping and bimanual pouring, at 4x speed.

Figures

From Figure 1: Overview of SALA. Left: unpaired human and robot videos are related only through language descriptions...
From Figure 1: Overview of SALA. Left: unpaired human and robot videos are related only through language descriptions of their atomic actions. Middle: supervising the latent action tokenizer with these relations organizes latent actions by action semantics rather than by scene. Right: more human-video pre-training lowers the robot's action prediction error, and more task-specific human video in post-training raises the real-world success rate.
From Figure 2: Method overview. (a) Latent action tokenizer: robot (blue) and human (orange) clips share the visual e...
From Figure 2: Method overview. (a) Latent action tokenizer: robot (blue) and human (orange) clips share the visual encoder, the SoftVQ quantizer and the visual decoder; only robot clips train the action decoder. Language similarities between atomic-action descriptions define soft neighborhoods that supervise the latent neighborhoods, with gradients stopped through robot latents. (b) VLA policy: the frozen tokenizer labels each clip with latent action tokens, which latent queries in the action expert learn to predict for both human and robot data; robot data also train the flow-matching action loss.
From Figure 4: Scaling with human video, averaged over four real-robot tasks. (a) Pre-training: validation L1 action ...
From Figure 4: Scaling with human video, averaged over four real-robot tasks. (a) Pre-training: validation L1 action error versus hours of human video, with the vanilla VLA (no latent actions) as a dashed line. (b) Post-training: success rate versus the fraction of each task's human demonstrations used, with the teleoperated demonstrations fixed.
Abstract

Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However, most existing LAMs learn latent actions solely through video reconstruction, which can cause their representations to cluster by visual appearance rather than by the underlying actions. As a result, they struggle to align human actions with their robot counterparts. To bridge the gap, we introduce SALA, which emphasizes establishing the semantic correspondence for latent actions. Without requiring paired data, we use the language descriptions of unpaired human and robot clips to estimate semantic similarities and supervise cross-embodiment correspondence. Specifically, we translate these similarities into soft neighborhood structures that guide human latent actions toward semantically related robot latents, while robot action reconstruction grounds the latent space in control. The resulting shared latent action space provides a unified supervision signal for training an embodied foundation model on both human and robot videos. Notably, latent action supervision enables scaling along two axes: (1) pre-training on increasing amounts of general human video consistently reduces robot action prediction error on downstream tasks, and (2) post-training on increasing amounts of task-specific human video substantially improves real-world robot task success rates.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.