Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations

Jason Jingzhou Liu, Tony Tao, Deepak Pathak, Ruslan Salakhutdinov

November 2026

TL;DR Pixel-UMI turns head-and-gripper human demonstrations into whole-body robot training data by retargeting them against a 3D reconstruction of each scene, so the policy learns postures that clear obstacles. Across five real mobile-manipulation tasks it reaches 87.0% average task progress, and scene-aware retargeting alone adds 64 points.

What they do

Handheld, UMI-style demonstrations record where the head and grippers go, but not how a mobile manipulator’s base, torso and arms should get there. Pixel-UMI converts human demonstrations, collected with grippers and cameras matched to the Pixel robot, into whole-body joint and base targets that respect the surrounding scene, and trains a policy that outputs those targets directly.

How it works

Environment-Aware Retargeting (EAR). A depth camera on the demonstrator’s headset reconstructs the static scene of each episode (ICP and pose-graph refinement of the Quest 3 tracking); objects that move are removed using free-space evidence. Geometric fabrics then track the demonstrated head and gripper poses while pushing the robot’s collision spheres away from the scene.
Delay-calibrated actions. Each joint’s tracking delay is measured on robot teleoperation data, and its action label looks ahead by that delay instead of using the next state.
REAM. Where retargeting cannot reproduce the demonstrated head or wrist pose, that camera’s tokens are masked during training with a probability that grows with the error’s percentile rank, so the policy relies on better-matched views.
DAgger. For contact-rich tasks, an operator corrects policy rollouts with VR controllers and the policy is retrained on the human demonstrations plus the corrected rollouts.

Results

20 trials per condition on five real tasks. EAR raises average task progress by 64.41 points over scene-unaware retargeting on Arrange Mouse, Pack Tennis Canister and Retrieve Drink; without it the elbow hits the box rim and Pack Tennis Canister scores zero.
Absolute joint targets reach 77.74%, against 25.12% for end-effector targets with a whole-body controller and 27.50% for relative joint targets. Calibrated labels add 24.05 points.
REAM brings the three tasks to 91.79%, 14.05 points above EAR alone.
On Fold Box and Hang Jacket, two DAgger rounds of 60 corrected rollouts each lift progress from 6.82% to 79.77%.

Limitations

State-only demonstrations do not reveal the forces applied during contact, which is why contact-rich tasks need robot corrections. A single head-mounted depth camera leaves occluded surfaces incomplete, the scene model is static, and only head and gripper motion is specified, so intentional contacts with an elbow or hip are not captured.

Figures

From Figure 2: Co-designed hardware. The sensorized human interface (left) and the Pixel robot (right) use matched gr... — From Figure 2: Co-designed hardware. The sensorized human interface (left) and the Pixel robot (right) use matched grippers and corresponding head and wrist camera layouts.

From Figure 4: Without EAR (left), retargeted robot postures intersect the surrounding scene, with collisions in red.... — From Figure 4: Without EAR (left), retargeted robot postures intersect the surrounding scene, with collisions in red. EAR (right) uses the reconstructed geometry to pick whole-body postures that avoid these obstacles while still tracking the demonstrated head and gripper motion.

From Figure 7: Human demonstrations and robot policy rollouts on the five tasks: Arrange Mouse, Pack Tennis Canister ... — From Figure 7: Human demonstrations and robot policy rollouts on the five tasks: Arrange Mouse, Pack Tennis Canister and Retrieve Drink test EAR, REAM and action assignment; Fold Box and Hang Jacket test on-policy correction.

From Figure 8: Task progress for (a) EAR versus retargeting without environment awareness and (b) end-effector target... — From Figure 8: Task progress for (a) EAR versus retargeting without environment awareness and (b) end-effector targets with a whole-body controller (EEF-Rel Base) versus relative and absolute joint targets. Both panels omit REAM.

Abstract

Sensorized human demonstrations offer scalable data for mobile manipulation, but embodiment differences complicate policy learning. We present Pixel-UMI, a pipeline for learning environment-aware whole-body policies from human data. We first introduce Environment-Aware Retargeting (EAR), which combines reconstructed scene geometry with demonstrated head and gripper trajectories to determine robot posture and generate collision-aware joint and base targets. Delay-calibrated action assignment converts these trajectories into actions. However, kinematic differences can leave residual pose errors, creating viewpoint mismatches between human demonstrations and robot observations. We therefore introduce Retargeting-Error-Aware Masking (REAM), which uses these errors during training to reduce reliance on mismatched views at deployment. Finally, because estimated actions cannot fully capture contact-rich interactions, we collect on-policy human corrections through DAgger to improve performance. We evaluate Pixel-UMI on five challenging tasks spanning articulated-object manipulation, deformable-object handling, constrained-space reaching, whole-body collision avoidance, and contact-rich bimanual coordination, achieving 87.0% average task progress. Video results are available at https://pixel-robot.com/pixel-umi

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong