Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Action Intent as a Bridge: Transferring Manipulation Knowledge from Human-Centric Videos to Robots

Zhiqian Xia, Lianyang Ma, Haifeng Xia, Siyu Xia, Wei Wang, Xiaochun Cao

November 2026

TL;DR AIT pretrains a VLA on human videos and robot data through a shared action intent (coarse motion primitives plus latent tokens stripped of scene noise), then decodes the intent into robot actions. On 12 real-robot tasks it reaches 82.1% success versus 70.4% for UniVLA.

What they do

Human videos are plentiful but have no action labels, and latent actions learned from them also absorb scene changes such as background and lighting. Action-Intent Transfer (AIT) bridges human and robot data with an action intent: a short sequence of motion primitives (move left, right, front, back, up or down; grasp; release) plus continuous latent tokens for the finer motion. A VLA is pretrained to predict this intent from both domains, then adapted to a target robot.

How it works

  • Explicit intents. Gemini 2.5 Pro labels human videos with an instruction and a primitive sequence, which annotators verify and correct. Robot trajectories get primitives from the dominant axis of end-effector displacement and from gripper changes.
  • Disentanglement. On DINOv2 features, an action tagger and a generator predict a future frame. Stage A, conditioned on the text and explicit intent, learns a codebook of scene tokens; stage B freezes it, so a second codebook of implicit intent tokens must explain the remaining motion.
  • Policy. A Prismatic-7B backbone is pretrained on HOI4D and TACO human videos plus Bridge-V2 to predict explicit, then implicit, intent tokens. LoRA fine-tuning on the target robot adds a lightweight action decoder trained with an L1 loss.

Results

  • Real robot (Astribot S1, 10-DoF right arm, 12 tasks, 20 trials each): 82.08% average success versus 70.42% for UniVLA with the same pretraining; 80.63% vs 73.13% on pick-up tasks and 85% vs 65% on the four spatial-direction tasks.
  • Adding human videos to pretraining lifts AIT on those four tasks from 73.8% to 85.0%, while UniVLA gains 5.0 points.
  • LIBERO: 91.3% average versus 88.1% for UniVLA. CALVIN ABC→D: 4.02 tasks completed in a row versus 3.80.
  • It runs at 8 Hz on one RTX 4090, against 10 Hz for UniVLA and 1.5 Hz for OpenVLA.

Limitations

The paper has no limitations section. Its text shows that the human-video labels need manual correction (before review, 82.4% of the VLM’s move primitives are correct), that the real-robot comparison covers only UniVLA on single-arm tabletop tasks, and that generating the intent costs 2 Hz relative to UniVLA.

Figures

From Figure 1: Three ways to learn from human videos. (a) MoCap-based methods need costly capture equipment and face ...
From Figure 1: Three ways to learn from human videos. (a) MoCap-based methods need costly capture equipment and face an embodiment gap. (b) Latent action learning uses raw videos, but its representations carry scene noise. (c) AIT separates scene noise from explicit and implicit action intents and decodes the intent into robot actions.
From Figure 2: Intent-guided latent disentanglement. A VLM writes instructions and explicit intents for human videos;...
From Figure 2: Intent-guided latent disentanglement. A VLM writes instructions and explicit intents for human videos; for robot data, explicit intents come from rules on end-effector displacement. An action tagger and a feature-wise generator on DINOv2 features then learn scene tokens (stage A) and, with those frozen, implicit intent tokens (stage B) by predicting a future frame.
From Figure 3: Policy alignment. A Prismatic-7B backbone predicts the explicit intent (e.g.
From Figure 3: Policy alignment. A Prismatic-7B backbone predicts the explicit intent (e.g. "move left, grasp") and then implicit intent tokens; at deployment an action decoder maps the implicit intent and the robot state to robot actions.
From Figure 4: Success rates on the 12 real-world tasks (T1-T12): AIT averages 82.08% versus 70.42% for UniVLA.
From Figure 4: Success rates on the 12 real-world tasks (T1-T12): AIT averages 82.08% versus 70.42% for UniVLA.
Abstract

Vision-Language-Action (VLA) models have achieved promising performance for generalized robotic manipulation, yet their reliance on expensive real-robot data remains a fundamental bottleneck. A scalable alternative is to transfer knowledge from off-the-shelf human-centric videos. However, existing methods extract latent action representations that are entangled with scene-specific noise, diluting the transferred knowledge. Motivated by this limitation, we propose Action-Intent Transfer (AIT), a two-stage method that transfers knowledge through a clean action representation shared across humans and robots. In the first stage, AIT extracts explicit action intents and disentangles the implicit action intent from manipulation videos. In the second stage, AIT aligns the human and robot domains within this shared intent space and decodes the intent into executable robot actions. Extensive experiments in both simulation and real-world environments demonstrate that AIT transfers human knowledge more effectively and consistently outperforms prior baselines.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.