VREgo: Consumer VR Headsets for Learning Robot Manipulation from Egocentric Demonstrations

Liuchuan Yu

November 2026

TL;DR Records manipulation demonstrations on a consumer VR headset (PICO 4 Ultra), using its own hand tracking and stereo cameras, and trains HumanEgo-style policies from them. In simulation, 40 demonstrations place a cube in 60-74 of 100 unseen layouts, close to Project Aria data, and a two-move stacking task rises from 2% to 93.8% after trajectory rewrites and execution rules.

What they do

Policies learned from egocentric human video usually use Project Aria glasses. Their head and hand poses come from Meta’s cloud processing, and their single camera needs a walk-up to the table to triangulate objects. VREgo records demonstrations on a consumer VR headset, the PICO 4 Ultra, which tracks the head and both hands on the device and has two see-through cameras 64 mm apart. It keeps HumanEgo’s representation and policy, adapts the preprocessing, and evaluates policies on a simulated WidowX arm in MuJoCo.

How it works

Capture. Object keypoints are triangulated from the stereo pair, so the wearer starts at the table. Repeated triangulations of static points agree within 1.1-4.3 mm, against 9-925 mm from head motion alone.
Preprocessing. A grasp threshold is fit per recording from the thumb-index distance, carried objects are relocated by the camera after each release, and SAM 2 propagates masks between keyframes.
Policy. The arm is inpainted out, a virtual gripper and object keypoints are drawn in, and a flow-matching policy predicts the hand pose and grasp for the next 50 frames.
Stacking fixes. Training trajectories are rewritten for a gripper (straight-down approach, pauses removed), and execution rules govern release timing, grasp centring and placement.

Results

Cube on tray, 40 takes: PICO policies succeed in 67, 74 and 60 of 100 unseen layouts over three seeds, Aria policies in 73, 78 and 0; the paired differences are not significant (p = 0.42 and 0.57). PICO data takes 1.3 GB against 15 GB.
Block on tray, then cube on its studs: the baseline places the block in 94 of 100 layouts but completes both moves in 2, mostly because replanning keeps deferring the release.
With rewritten trajectories and all execution rules, both moves succeed in 93.8 of 100 layouts (mean over four noise seeds).

Limitations

All results are in simulation, with simulator-supplied masks and arm removal, so deployed perception is untested. One operator recorded both tasks in one room. The execution rules use task-specific information and simulator object positions, and were designed on the same 100 layouts used to score them.

Figures

From Figure 1: Left, one frame of a two-move demonstration (red block onto the tray, then green cube onto the block's... — From Figure 1: Left, one frame of a two-move demonstration (red block onto the tray, then green cube onto the block's studs) as the headset's two see-through cameras recorded it. Middle, the same frame as a training image, with the arm removed, a virtual gripper drawn at the tracked hand and keypoints on the tray, block and cube. Right, in simulation, a policy trained on 40 demonstrations sets the cube on the block in an unseen layout.

From Figure 3: One stacking demonstration through the preprocessing at five moments. Top: the left-eye image. Middle:... — From Figure 3: One stacking demonstration through the preprocessing at five moments. Top: the left-eye image. Middle: detection masks of the arm (red), tray (cyan), block (green) and cube (magenta). Bottom: the training image, with the arm removed, a virtual gripper at the tracked hand and each object's keypoints, which follow the block onto the tray and the cube onto the block.

From Figure 4: Unseen layouts (of 100) in which the policy set its object down, against the number of training takes:... — From Figure 4: Unseen layouts (of 100) in which the policy set its object down, against the number of training takes: cube on the tray from PICO and Aria recordings, and the stacking task's first move (block on the tray) and both moves. Bars at 40 takes span the cube task's training seeds.

From Figure 6: Layouts (of 100) in which the 40-take stacking policy set the block on the tray and completed both mov... — From Figure 6: Layouts (of 100) in which the 40-take stacking policy set the block on the tray and completed both moves: the original policy, then the policy retrained on rewritten records with execution rules added one at a time. Points are means over four noise seeds.

Abstract

VREgo trains manipulation policies from demonstrations recorded on consumer VR headsets; we instantiate it with EgoKit on one such headset, the PICO 4 Ultra. On-device head and hand tracking and stereo object localisation allow the operator to begin a demonstration at the table. We adapt HumanEgo's preprocessing to estimate grasp thresholds per recording, update object positions after release, and propagate segmentation masks between keyframes. We evaluate policies in simulation on two tabletop tasks recorded by one operator. For cube placement, we compare PICO and Project Aria recordings of the same 41 layouts. With 40 training demonstrations, the PICO policies placed the cube on the tray in 67, 74, and 60 of 100 evaluation layouts across three training seeds. The corresponding Aria results were 73, 78, and 0. For a two-stage stacking task recorded with the PICO alone, the baseline placed the block on the tray in 94 of 100 layouts and met the end-of-episode criterion for both placements in two. In one diagnostic episode, repeated replanning deferred release beyond the eight executed steps. Demonstration replays also showed that the open gripper could displace the cube before grasping it. After adapting the training trajectories and adding task-specific execution rules that use simulator object positions, the system met both placement criteria in 93.8% of episodes across four evaluation noise seeds. The same 100 layouts informed rule development and evaluation. VREgo is available at https://vrego.chuange.org/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong