Lift, See, Act: Hierarchical Robot Policy Pretraining with 3D Foundation Models
TL;DR Pretrains a robot policy's visual encoder on human hand videos by distilling the features and depth maps of 3D foundation models and predicting 3D wrist trajectories. With the encoder frozen, a Franka policy averages 68.3% success across six real-world task settings, against 41.7% for the best baseline.
What they do
Human videos are a cheap source for pretraining a robot’s visual encoder, but they carry no action labels. Recent methods run 3D foundation models (3D FMs) to estimate 3D hand trajectories and keep only the trajectories. LSA (Lift, See, Act) also keeps what the 3D FMs compute along the way, dense features and depth maps, and uses them as extra supervision. The encoder is then frozen under a small behavior-cloning head.
How it works
- Data engine. On 20K cleaned DexYCB clips, VGGT estimates camera intrinsics and Depth Pro metric depth, which lift the detected 2D wrist into a 3D trajectory.
- Lift. Layer-3 tokens of a LoRA-adapted ViT-B are aligned with the VGGT and Depth Pro features, with per-teacher weights computed from the encoder’s class tokens.
- See. A decoder reconstructs the metric depth map from layer-6 tokens. The attention head that best matches a hand-keypoint heatmap becomes the target for the averaged attention of the first six layers.
- Act. A one-layer MLP predicts the next three 3D wrist positions.
Results
- MetaWorld (8 tasks): LSA is best or tied on every task, averaging 92.0% against 87.6% for H-RDT and 82.6% for VITRA. The same holds on all five DexArt and Adroit dexterous-hand tasks.
- Ablation on four MetaWorld tasks: 73.3% without human-video pretraining, 81.0% with the 3D trajectory objective alone, 92.2% for full LSA. Feeding frozen 3D-FM features straight to the policy reaches at most 40.7% and runs at least 3.9x slower.
- Franka Research 3 (30 teleoperated demos per task, 10 trials): 68.3% average over four tasks and two unseen towel colors, against 41.7% for HRP; 70% and 60% with the unseen towels, where baselines reach at most 20% and 40%.
Limitations
Only the 3D wrist trajectory is supervised; adding MANO hand-mesh targets hurt in preliminary tests, which the authors attribute to the mismatch between fingers and a parallel gripper. Attentive guidance gives only coarse region-level cues, and scene differences between human videos and robot workspaces may still limit transfer.
Videos
Figures



Abstract
Robot policy pretraining based on human videos is crucial for improving the policy's generalization ability. One main challenge for this task is the lack of explicit action-relevant representations in such unlabeled data. Recent works tend to estimate 3D hand motion trajectories from videos using 3D foundation models (3D FMs). However, they keep solely the sparse trajectories as supervision, ignoring the rich, fine-grained information produced during the trajectory extraction process. To address this issue, we propose LSA, a hierarchical robot policy pretraining framework that includes three phases: Lift, See, and Act. Specifically, the Lift phase elevates 2D video observations into dense 3D representations by adaptively aligning features in the policy's shallow layers with the intermediate representations extracted from multiple 3D FMs. Building upon these lifted representations, the See phase equips the policy with explicit geometric and interactive awareness through dual guidance. It introduces depth-map reconstruction to help the model comprehend 3D spatial layouts and utilizes hand region cues to explicitly supervise the encoder's attention toward task-relevant interaction hotspots. Empowered by the dense 3D representation learning and precise spatial guidance, our policy achieves robust and accurate robotic manipulation in the Act phase. Extensive experiments on diverse simulation and real-world tasks demonstrate that LSA significantly outperforms current state-of-the-art approaches. The project page is available at: https://yiyuan00.github.io/Lift-See-Act.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
