← Paper Digest

Ego2X: Contact-aware Transfer from Egocentric Human Data to Any Robot

Xinke Lyu, Bowen Yang, Zhicheng Wang

November 2026

TL;DR Ego2X turns monocular egocentric human video into robot training data by transferring object-frame contacts instead of hand motion, for dexterous hands and parallel grippers. Post-training X-VLA on its converted episodes alone, with no simulator demos, raises success in RoboTwin 2.0 from 40.5% to 67.0%.

What they do

Pipelines that retarget egocentric human video to robots usually transfer hand motion. Monocular hand estimates drift by centimeters along the camera ray, so retargeted fingers hover beside objects and grippers close on empty space. Ego2X transfers what happens on the object instead: where it is touched, how hard, with which grasp and in which phase. From one monocular video and a robot URDF it outputs a LeRobot episode with robot actions and a video in which the human arm is replaced by the robot.

How it works

  • CORE contact records. Each contact is stored in the object frame: a surface anchor and normal (camera rays intersected with the object’s SDF, timed by the EgoTac contact detector), a relative force prior, one of seven grasp modes, and an interaction state from approach to release. Anchors are graded by how much evidence supports them.
  • Depth correction. Object and hand are corrected on separate paths to avoid circular scale coupling; the hand is shifted along its camera ray to fit trusted anchors.
  • Contact-driven IK. A URDF compiler turns robot collision meshes into contact pads. IK pulls dexterous-hand pads onto the anchors; for parallel grippers, one antipodal object-frame grasp is locked per contact interval.
  • Compositing and verification. The held object is protected while the arm is inpainted, and episodes are replayed in RoboTwin 2.0; only physically successful ones are kept.

Results

  • HOT3D: tactile-seeded ray anchoring cuts contact anchor error from 6.69 to 1.96 mm; with HaWoR hands, in-contact ray depth error drops 33.7%.
  • EgoDex (79 takes): 62.9% of annotated contacts realized vs. 27.9% for HuRo; parallel-gripper two-pad fit 63.2% vs. 9.7% for Ego2Robot.
  • Simulation conversion yield: 31/52 bread-in-basket videos vs. 1/52 for HuRo and Ego2Robot; 45/53 vs. 0/53 for bell-on-scale.
  • X-VLA trained on Ego2X episodes only: bread-in-basket success goes from 40.5% to 67.0% over 200 scenes (20 simulator demos give 72.0%); bell-on-scale from 17.0% to 55.0%.

Limitations

All robot results are in simulation; the authors say physical deployment is still underway. Contact localization suffers when object pose estimation fails under heavy occlusion, and forces are visual-kinematic priors rather than calibrated Newtons.

Figures

From Figure 1: Ego2X transfers object-frame contacts rather than hand motion. From monocular video it (a) recovers ob...
From Figure 1: Ego2X transfers object-frame contacts rather than hand motion. From monocular video it (a) recovers object-frame CORE contact records, (b) uses them for depth alignment, contact IK, robot compositing and simulation transfer, and (c) beats prior pipelines in closed-loop simulation conversion (59.6% vs. 1.9%) and X-VLA success rate (67.0% vs. 40.5% zero-shot).
From Figure 2: Ego2X pipeline and CORE overview. From monocular video, Ego2X reconstructs metric hands and signed obj...
From Figure 2: Ego2X pipeline and CORE overview. From monocular video, Ego2X reconstructs metric hands and signed object surfaces, extracts object-frame CORE contact records (contact point, force, grasp mode, interaction state), and uses them for depth correction, cross-morphology retargeting, de-skinned compositing and closed-loop simulation verification.
From Figure 7: Closed-loop verification and X-VLA policy learning. (a) Place bread in basket over 200 paired scenes, ...
From Figure 7: Closed-loop verification and X-VLA policy learning. (a) Place bread in basket over 200 paired scenes, (b) Place bell on scale ablations, (c) demonstration yield on matched videos, (d) co-training with simulator demos.
Abstract

Embodied models depend on real interaction data, yet collecting it on physical robots is slow and labor-intensive. Recent ego-to-robot pipelines retarget first-person view human videos to robot data via transferring hand and arm motion. They judge fidelity mainly by visual appearance. However, trajectories often fail in terms of physics: fingers hover beside objects, and grippers close on empty space. We argue that how the object moves and the contacts that drive the motion count. Thus, we present Ego2X, an object-centric framework that converts monocular egocentric video into physically executable robot data across dexterous hands and parallel grippers. Ego2X centers on CORE (Contact-centric Object-interaction REpresentation), which records each contact on the object by its surface anchor, as well as relative force, grasp mode, and interaction state. Throughout the pipeline, CORE corrects hand-object drift via object-relative constraints, steers inverse kinematics toward contacts the robot must realize, and preserves held objects during arm erasure and robot compositing. On HOT3D, contact-driven correction reduces in-contact hand depth error by 33.7%, while on EgoDex, contact-driven retargeting realizes 62.9% of annotated contacts, surpassing the official HuRo pipeline (27.9%). In RoboTwin 2.0, post-training X-VLA on Ego2X episodes alone, without simulator demonstrations, raises task success rate from 40.5% zero-shot to 67.0% across 200 paired scenes. We will publicly release our conversion pipeline and CORE-annotated dataset for dexterous hands and parallel grippers.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.