← Paper Digest

SHAPE: Discrete Supervision and Parallel Continuous Streams for Learning Dexterous Manipulation from Egocentric Human Data

Yaping Li, Zhihong Liu, Jiacheng Li, Yi Yang, Wenchao Yue, Mingzhe Ye, Xiangcheng Liu, Yiyang Chen, Haotian He, Qi Zhang, Haonan Wang, Dahua Lin, Zhijie Deng

November 2026

TL;DR SHAPE is a dexterous VLA that lets egocentric human video supervise its backbone through discrete end-effector action tokens, and outputs both end-effector and joint actions. On a bimanual robot with 22-DoF hands it scores 0.80 across six tasks against 0.63 for π0.5, and learns new tasks from human demonstrations alone at 35% success.

What they do

Egocentric human recordings are much cheaper than dexterous robot demonstrations, but give only retargeted end-effector (EEF) poses, not the joint commands a robot hand needs. SHAPE, a vision-language-action model for a dual-arm robot with two 22-DoF hands, addresses how human data should supervise the policy and which action space to execute.

How it works

  • Data. A Qwen3-VL-4B backbone is pretrained on 2.8k hours of egocentric human recordings and 228 hours of robot teleoperation, sampled 1:1.
  • Discrete supervision. The backbone predicts FAST tokens of each EEF action chunk for human and robot samples alike, so human data reaches the backbone directly, not only through the action decoder. The tokens are dropped at deployment.
  • Two executable streams. A shared 0.6B flow-matching expert outputs an EEF stream (wrist pose and fingertips, turned into joint targets by IK) and a joint stream trained on robot data only. An attention mask keeps them independent, so either can drive the robot.

Results

  • Fine-tuned on robot demonstrations, the joint stream scores 0.80 across six real tasks (20 trials each), against 0.63 for π0.5 and 0.51 for GR00T N1.7. The same model pretrained without human data scores 0.64.
  • Fine-tuned on human demonstrations only, SHAPE succeeds in 21/60 trials (35%) on three tasks, against 7/60 without FAST pretraining and 0-4/60 with robot-only pretraining. Joint readout never succeeds here (0/240 across models).
  • On 24 simulated RoboCasa tasks with the GR-1 humanoid, absent from pretraining, it reaches 72.2% against 64.8% for π0.5.

Limitations

Human-to-robot transfer reaches 35%; most failures are grasps that close too loosely, from morphology gaps that retargeting does not bridge. Millimeter-tolerance connector reinsertion stays hard (2/20 full reinsertions per connector). Each stream is better in only one regime: joint readout with robot demonstrations, EEF readout with human ones. With robot data alone, FAST tokens raise error on held-out robot tasks.

Figures

From Figure 1: (a) Pretraining pairs egocentric human recordings, which give only retargeted end-effector (EEF) actio...
From Figure 1: (a) Pretraining pairs egocentric human recordings, which give only retargeted end-effector (EEF) actions, with robot teleoperation (EEF and joint actions). (b) The backbone predicts discrete EEF tokens as an auxiliary target, while a shared flow-matching expert outputs separate EEF and joint streams. (c) Gains from human data and discrete tokens under robot-demo and human-only fine-tuning, and which stream suits each.
From Figure 2: Overview of SHAPE. Human recordings give head images and EEF actions; robot data add wrist images and ...
From Figure 2: Overview of SHAPE. Human recordings give head images and EEF actions; robot data add wrist images and joint actions. The backbone predicts FAST tokens of the EEF chunk, the sub-task and future frames, while a shared flow-matching expert denoises an EEF chunk and a joint chunk kept apart by the attention mask. At deployment the training-only heads are dropped and either stream can drive the robot.
From Figure 3: The robot in top and side view, and the four stages of each of the six real-robot tasks fine-tuned wit...
From Figure 3: The robot in top and side view, and the four stages of each of the six real-robot tasks fine-tuned with robot demonstrations, with partial credit in parentheses for Reinsert USB and Reinsert Type-C.
From Figure 4: Human-to-robot transfer with no task-specific robot demonstrations. Top: egocentric human demonstratio...
From Figure 4: Human-to-robot transfer with no task-specific robot demonstrations. Top: egocentric human demonstrations, the only fine-tuning data. Bottom: SHAPE robot rollouts with the EEF stream.
Abstract

Egocentric human recordings are far cheaper than dexterous robot demonstrations, promising stronger pretraining and direct skill learning without task-specific robot data. Yet humans and robots differ in morphology and kinematics, so human recordings lack the joint positions and actuator commands that supervise robot policies, providing only retargeted end-effector (EEF) actions. This raises two questions: how human data should supervise the policy, and how a dexterous robot should read out actions at deployment. We address both in SHAPE (Supervision from Human Action tokens with Parallel Executable streams), a vision-language-action (VLA) model whose backbone predicts discrete action tokens of EEF chunks as an auxiliary task, while a shared flow-matching expert outputs two executable streams, EEF and joint. On a bimanual robot with 22-DoF hands, a matched study shows that discrete supervision helps learn from human data, driving backbone gradients and shaping action-informative representations across domains. Under standard control pipelines at deployment, direct joint readout excels at fine-grained control with robot demonstrations, scoring 0.80 across six tasks against 0.63 for π0.5, whereas IK-based EEF readout excels at human-to-robot transfer. Pretrained with human data and discrete tokens, SHAPE learns new tasks from human demonstrations without task-specific robot data.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.