← Paper Digest

Lift, See, Act: Hierarchical Robot Policy Pretraining with 3D Foundation Models

Yiyuan Ge, Changxing Ding, Ziyu Hao, Zijie Zheng, Xiangmin Xu

November 2026

Wipe PC: the Franka picks up a pink towel and wipes the laptop.

TL;DR Pretrains a robot policy's visual encoder on human hand videos by distilling the features and depth maps of 3D foundation models and predicting 3D wrist trajectories. With the encoder frozen, a Franka policy averages 68.3% success across six real-world task settings, against 41.7% for the best baseline.

What they do

Human videos are a cheap source for pretraining a robot’s visual encoder, but they carry no action labels. Recent methods run 3D foundation models (3D FMs) to estimate 3D hand trajectories and keep only the trajectories. LSA (Lift, See, Act) also keeps what the 3D FMs compute along the way, dense features and depth maps, and uses them as extra supervision. The encoder is then frozen under a small behavior-cloning head.

How it works

  • Data engine. On 20K cleaned DexYCB clips, VGGT estimates camera intrinsics and Depth Pro metric depth, which lift the detected 2D wrist into a 3D trajectory.
  • Lift. Layer-3 tokens of a LoRA-adapted ViT-B are aligned with the VGGT and Depth Pro features, with per-teacher weights computed from the encoder’s class tokens.
  • See. A decoder reconstructs the metric depth map from layer-6 tokens. The attention head that best matches a hand-keypoint heatmap becomes the target for the averaged attention of the first six layers.
  • Act. A one-layer MLP predicts the next three 3D wrist positions.

Results

  • MetaWorld (8 tasks): LSA is best or tied on every task, averaging 92.0% against 87.6% for H-RDT and 82.6% for VITRA. The same holds on all five DexArt and Adroit dexterous-hand tasks.
  • Ablation on four MetaWorld tasks: 73.3% without human-video pretraining, 81.0% with the 3D trajectory objective alone, 92.2% for full LSA. Feeding frozen 3D-FM features straight to the policy reaches at most 40.7% and runs at least 3.9x slower.
  • Franka Research 3 (30 teleoperated demos per task, 10 trials): 68.3% average over four tasks and two unseen towel colors, against 41.7% for HRP; 70% and 60% with the unseen towels, where baselines reach at most 20% and 40%.

Limitations

Only the 3D wrist trajectory is supervised; adding MANO hand-mesh targets hurt in preliminary tests, which the authors attribute to the mismatch between fingers and a parallel gripper. Attentive guidance gives only coarse region-level cues, and scene differences between human videos and robot workspaces may still limit transfer.

Videos

Pick Trash: the robot picks up a piece of trash and drops it into the bin.
Wipe PC with a purple towel, a color not seen in training.

Figures

From Figure 1: (a) Existing methods pretrain on human videos using only the estimated hand motion trajectories as sup...
From Figure 1: (a) Existing methods pretrain on human videos using only the estimated hand motion trajectories as supervision. (b) LSA also uses the intermediate features and depth maps of several 3D foundation models.
From Figure 2: (a) The data engine uses two 3D foundation models for depth maps and camera intrinsics, lifts 2D wrist...
From Figure 2: (a) The data engine uses two 3D foundation models for depth maps and camera intrinsics, lifts 2D wrist points into 3D hand trajectories, and keeps dense geometric features. (b) Pretraining aligns the shallow encoder layers with those features, reconstructs depth, guides attention with hand heatmaps and predicts the 3D hand trajectory. (c) The frozen encoder and a policy head run closed-loop robot control.
From Figure 4: The real-world setup: robot and sensors (left), the four tasks in the in-distribution setting (middle)...
From Figure 4: The real-world setup: robot and sensors (left), the four tasks in the in-distribution setting (middle), and the changes made for the out-of-distribution tests (right).
Abstract

Robot policy pretraining based on human videos is crucial for improving the policy's generalization ability. One main challenge for this task is the lack of explicit action-relevant representations in such unlabeled data. Recent works tend to estimate 3D hand motion trajectories from videos using 3D foundation models (3D FMs). However, they keep solely the sparse trajectories as supervision, ignoring the rich, fine-grained information produced during the trajectory extraction process. To address this issue, we propose LSA, a hierarchical robot policy pretraining framework that includes three phases: Lift, See, and Act. Specifically, the Lift phase elevates 2D video observations into dense 3D representations by adaptively aligning features in the policy's shallow layers with the intermediate representations extracted from multiple 3D FMs. Building upon these lifted representations, the See phase equips the policy with explicit geometric and interactive awareness through dual guidance. It introduces depth-map reconstruction to help the model comprehend 3D spatial layouts and utilizes hand region cues to explicitly supervise the encoder's attention toward task-relevant interaction hotspots. Empowered by the dense 3D representation learning and precise spatial guidance, our policy achieves robust and accurate robotic manipulation in the Act phase. Extensive experiments on diverse simulation and real-world tasks demonstrate that LSA significantly outperforms current state-of-the-art approaches. The project page is available at: https://yiyuan00.github.io/Lift-See-Act.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.