Ego2X: Contact-aware Transfer from Egocentric Human Data to Any Robot

TL;DR Ego2X turns monocular egocentric human video into robot training data by transferring object-frame contacts instead of hand motion, for dexterous hands and parallel grippers. Post-training X-VLA on its converted episodes alone, with no simulator demos, raises success in RoboTwin 2.0 from 40.5% to 67.0%.
What they do
Pipelines that retarget egocentric human video to robots usually transfer hand motion. Monocular hand estimates drift by centimeters along the camera ray, so retargeted fingers hover beside objects and grippers close on empty space. Ego2X transfers what happens on the object instead: where it is touched, how hard, with which grasp and in which phase. From one monocular video and a robot URDF it outputs a LeRobot episode with robot actions and a video in which the human arm is replaced by the robot.
How it works
- CORE contact records. Each contact is stored in the object frame: a surface anchor and normal (camera rays intersected with the object’s SDF, timed by the EgoTac contact detector), a relative force prior, one of seven grasp modes, and an interaction state from approach to release. Anchors are graded by how much evidence supports them.
- Depth correction. Object and hand are corrected on separate paths to avoid circular scale coupling; the hand is shifted along its camera ray to fit trusted anchors.
- Contact-driven IK. A URDF compiler turns robot collision meshes into contact pads. IK pulls dexterous-hand pads onto the anchors; for parallel grippers, one antipodal object-frame grasp is locked per contact interval.
- Compositing and verification. The held object is protected while the arm is inpainted, and episodes are replayed in RoboTwin 2.0; only physically successful ones are kept.
Results
- HOT3D: tactile-seeded ray anchoring cuts contact anchor error from 6.69 to 1.96 mm; with HaWoR hands, in-contact ray depth error drops 33.7%.
- EgoDex (79 takes): 62.9% of annotated contacts realized vs. 27.9% for HuRo; parallel-gripper two-pad fit 63.2% vs. 9.7% for Ego2Robot.
- Simulation conversion yield: 31/52 bread-in-basket videos vs. 1/52 for HuRo and Ego2Robot; 45/53 vs. 0/53 for bell-on-scale.
- X-VLA trained on Ego2X episodes only: bread-in-basket success goes from 40.5% to 67.0% over 200 scenes (20 simulator demos give 72.0%); bell-on-scale from 17.0% to 55.0%.
Limitations
All robot results are in simulation; the authors say physical deployment is still underway. Contact localization suffers when object pose estimation fails under heavy occlusion, and forces are visual-kinematic priors rather than calibrated Newtons.
Figures



Abstract
Embodied models depend on real interaction data, yet collecting it on physical robots is slow and labor-intensive. Recent ego-to-robot pipelines retarget first-person view human videos to robot data via transferring hand and arm motion. They judge fidelity mainly by visual appearance. However, trajectories often fail in terms of physics: fingers hover beside objects, and grippers close on empty space. We argue that how the object moves and the contacts that drive the motion count. Thus, we present Ego2X, an object-centric framework that converts monocular egocentric video into physically executable robot data across dexterous hands and parallel grippers. Ego2X centers on CORE (Contact-centric Object-interaction REpresentation), which records each contact on the object by its surface anchor, as well as relative force, grasp mode, and interaction state. Throughout the pipeline, CORE corrects hand-object drift via object-relative constraints, steers inverse kinematics toward contacts the robot must realize, and preserves held objects during arm erasure and robot compositing. On HOT3D, contact-driven correction reduces in-contact hand depth error by 33.7%, while on EgoDex, contact-driven retargeting realizes 62.9% of annotated contacts, surpassing the official HuRo pipeline (27.9%). In RoboTwin 2.0, post-training X-VLA on Ego2X episodes alone, without simulator demonstrations, raises task success rate from 40.5% zero-shot to 67.0% across 200 paired scenes. We will publicly release our conversion pipeline and CORE-annotated dataset for dexterous hands and parallel grippers.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
