Ego Guidance: Enhancing Robot Policy Learning with Action-Free Human Egocentric Videos

Lin Long, Tianren Ma, Zhong Chu, Di Wu, Maosong Cao, Qixiang Ye, Junbo Zhao, Shanghang Zhang

November 2026

TL;DR SPAR adds a future-visual-feature prediction head to a VLA so that action-free human egocentric video can train the shared VLM backbone. LIBERO-Plus success rises as video grows from 0 to 1,500 hours, and on five real bimanual tasks it scores 20/50 against 22/50 for pi0.5.

What they do

Human egocentric video is plentiful but mostly has no action labels. The paper asks whether it can still help a robot policy without recovering actions. Its model, SPAR, trains a VLM backbone on two prediction tasks at once: robot actions and the features of a future camera frame. Human video supervises only the second, so any benefit to the policy has to pass through the shared backbone.

How it works

Two experts on one VLM. An action expert (flow matching) and a generation expert each cross-attend to the per-layer hidden states of Qwen3.5-2B and never attend to each other. The generation expert is removed at inference, so the deployed policy costs the same as a standard VLA.
Prediction target. One future frame of the primary camera, encoded by a frozen visual encoder (SigLIP2 by default) and aligned with the end of the action chunk.
Two-stage training. Co-train on robot data (about 230 hours curated from DROID) and egocentric video, then fine-tune on downstream demonstrations alone.
Video curation. Ego4D clips are kept only if camera motion is stable and a hand is visible, leaving 975 of 3,976 hours; a VLM writes a step description for each 5-second clip.

Results

Target choice matters on LIBERO-Plus: SigLIP2 76.2%, Qwen-ViT 75.6%, DINOv2 74.9%, against 60.0% for the reconstruction-oriented Wan-VAE latent. Predicting four frames instead of one changes the average by 0.30 points.
Raw Ego4D clips underperform robot-only training; clips passing both gates beat it at 16k and 20k fine-tuning steps. With 0 to 1,500 hours of Open-AoE video, LIBERO success changes little while LIBERO-Plus success rises steadily.
Real robot (AgileX Cobot Magic, five tasks, 10 trials each): 20/50 against 22/50 for pi0.5. Under visual perturbations it beats pi0.5 on Cucumber Transfer (9, 8, 5 against 7, 7, 4 out of 10) and trails it on Flower Handover.

Limitations

Both methods rarely complete Cup Hanging, Bottle Uncapping or Toaster Loading (0-2 of 10), and most of SPAR’s failures occur during grasping. The robustness gain under perturbations is task-dependent. The authors call the diversity study preliminary; it showed no consistent gain. The real-robot model used a larger robot corpus (about 8,000 hours) than the simulation studies.

Figures

From Figure 1: Architecture overview of SPAR. Starting from a common VLA, SPAR extends it with a generation expert th... — From Figure 1: Architecture overview of SPAR. Starting from a common VLA, SPAR extends it with a generation expert that mirrors the action expert design, and predicts the representation of the future visual state simultaneously. Robot data supervises both experts, while the egocentric videos supervise only the generation expert. The VLM backbone can therefore be optimized with heterogeneous data composition. The generation expert is removed at inference.

From Figure 3: Ego4D curation and corresponding OOD task success rates. Left: hours retained after the motion-quality... — From Figure 3: Ego4D curation and corresponding OOD task success rates. Left: hours retained after the motion-quality and visible-hand gates. Right: robot-only training compared with co-training on raw Ego4D, Gate-1 clips, or clips passing both gates, across fine-tuning steps.

From Figure 4a: Success on LIBERO (in-domain) and LIBERO-Plus (OOD) as the amount of egocentric video in pretraining ... — From Figure 4a: Success on LIBERO (in-domain) and LIBERO-Plus (OOD) as the amount of egocentric video in pretraining grows from 0 to 1,500 hours.

From Figure 6: Real-world generalization settings for Cucumber Transfer and Flower Handover: default, illumination ch... — From Figure 6: Real-world generalization settings for Cucumber Transfer and Flower Handover: default, illumination changes, background variations and static distractors.

Abstract

In this paper, we investigate how human egocentric videos can benefit robot policy learning without action supervision. Building on a pretrained VLM, we propose Shared Prediction on Action and Representation (SPAR), an MoT architecture in which future visual state prediction can provide independent and complementary learning signals alongside action prediction. We find that representations learned from egocentric videos improve downstream performance on out-of-distribution robot tasks, with further gains as video data scales. We also study the effects of objective choices and data curation, providing insights into how to leverage action-free videos for scalable robot learning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong