Ego2Sim2Humanoid: Agentic Real-to-Sim for Learning Loco-Manipulation VLAs without Robot Demonstrations
- arXiv
- Ask alphaXiv
- Project page
- Humanoids and mobile manipulation
- Imitation learning and data collection

TL;DR E2S2H turns recordings from egocentric smart glasses into physically checked whole-body training data for a Unitree G1 humanoid, then fine-tunes the GR00T N1.7 VLA on that data alone. Without any real-robot demonstrations, the policy lifts a bottle in 9 of 20 real trials and puts on a hat in 5 of 20.
What they do
Humanoid VLAs are usually trained on teleoperated robot demonstrations. E2S2H uses recordings from Project Aria Gen 2 glasses instead, which capture RGB-D video, the head trajectory and in-view hand poses but not the body. Each recording becomes a physically verified whole-body trajectory for a Unitree G1 with three-fingered Dex3 hands, and a VLA is trained on that data alone.
How it works
- Agentic real-to-sim. A coding agent reconstructs each recording without a pre-scan: scene and object meshes, object tracks, completed unseen surfaces, and a target contact band on each grasped object.
- Motion adaptation. A grasp planner maps the contact band to a Dex3 wrist pose and finger closure, bilevel optimization picks collision-free stances, and a motion generator fills in locomotion and out-of-view arm motion.
- Correction in physics. After kinematic smoothing, an RL tracking policy executes the motion in simulation under agent-written task rewards, which also cover phases the camera cannot see.
- Robot-view rendering. Rendering the scene through the G1’s head camera along each accepted rollout gives synchronized images, proprioception and actions for fine-tuning GR00T N1.7.
Results
- On 71 hat recordings, the grasp planner cuts median finger penetration from 1.8 to 0.8 cm compared with copying the human palm.
- Against EgoHumanoid and EgoHumanoid-V2, the robot ends 0.8 cm from the contact region (vs 70.8-112.9 cm) and yields 57 feasible rim grasps out of 71 (vs 0).
- On the real G1, with no robot demonstrations, the policy lifts the bottle in 9 of 20 trials and puts the hat on in 5 of 20. Most failures happen at the grasp.
Limitations
Object mass and friction are guessed from priors, so heavy objects may not transfer, and rigid or hinged object models limit it on deformable tasks such as folding clothes. Thin objects leave the grasp planner little margin: only 59 of 154 hat recordings survive the pipeline.
Figures




Abstract
Imitation learning for visuomotor policies typically relies on teleoperated robot demonstrations, which are costly because every episode requires a trained operator controlling a physical robot in real time through teleoperation interfaces and wearable body-tracking hardware. Egocentric glasses capture human demonstrations without a robot or handheld grippers, yet they track only the head and in-view human hands, and directly copying human motion to a robot with different stature, arm reach and hand morphology fails to produce feasible contacts. E2S2H converts egocentric glasses recordings into physically verified robot training trajectories. An agentic real-to-sim stage reconstructs collision and textured visual assets from the recorded RGB-D and head streams, completes unseen surfaces when needed, and predicts target contact bands on the manipulated objects. During motion adaptation, a grasp planner turns each contact band into a feasible robot wrist pose and finger closure, bilevel stance optimization plans collision-free standing postures that keep hand targets reachable, and a whole-body motion generator synthesizes lower-body locomotion and out-of-view arm motion. Two-stage motion correction then blends visibility transitions kinematically and trains a reinforcement-learning tracking policy in physics simulation under agent-designed semantic rewards, completing manipulation phases even when the object leaves the camera view. Rendering the reconstructed scene through the robot's onboard camera along the optimized rollout yields synchronized robot-view images, proprioception and verified action labels. Fine-tuned exclusively on this converted dataset, without real-robot demonstrations, GR00T N1.7 performed loco-manipulation tasks autonomously on a Unitree G1. Project page: https://robonana.github.io/ego2sim2humanoid/
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
