EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming

Yaru Niu, Binghong Chen, Changwei Yao, Shangtao Li, Zhenlong Fang, Revanth Krishna Senthilkumaran, Shuai Zhou, Yuemin Mao, Kalie Ching, Po-Ju Chao, Yudong Luo, Peiqi Yu, Heng Zhang, Hao Zhang, Bingqing Chen, Chen Qiu, H. Eric Tseng, Ye Yuan, Umar Iqbal, Changliu Liu, Jonathan Francis, Ding Zhao

November 2026

The G1 humanoid inserts the light-purple T-block used in its fine-tuning demonstrations.

TL;DR EgoX pretrains one modular Transformer policy on egocentric demonstrations from humans, five real robots and a simulated humanoid, then fine-tunes it on each target robot. Pretraining raises average task progress from 0.63 to 0.80 across 13 task-robot pairs, and from 0.53 to 0.77 out of distribution.

What they do

Humans, robots and simulated agents produce demonstrations with different bodies, cameras and action spaces. EgoX treats them all as embodiments of equal standing: it pretrains one policy, MXT+, on their combined egocentric data and then fine-tunes it on a target robot and task. The data cover a human, a Unitree G1 humanoid with five-finger hands, an xArm7 with a LEAP Hand, a LocoMan quadruped, a Galaxea R1 Lite, a Unitree Z1 with a fixed scoop and a simulated G1: 4,851 trajectories, 34.38 hours.

How it works

Shared modalities. Observations and actions are split into corresponding parts (egocentric image, torso pose, left/right wrist poses, end-effector keypoints) in common root, egocentric and wrist frames. Fingertips, gripper contacts and tool points become comparable keypoints.
Modular Transformer. Each embodiment has its own tokenizer per observation modality and its own expert per action modality, joined by a shared encoder-decoder trunk that predicts 2 s action chunks.
Embodiment dreaming. A training-only head predicts latent tokens of future images and joint states, with targets from EMA copies of the tokenizers. Deployment does not use it.
Pretraining draws an equal-sized batch from every embodiment and holds out the target robot’s target-task data for fine-tuning.

Results

Pretraining raises average task progress from 0.63 to 0.80 (out of distribution 0.53 to 0.77). Pretrained MXT+ ranks first among four methods, including ACT and pi0.5, on 12 of 13 pairs, with 20 trials per real-robot pair.
Broader mixtures transfer better: the full mixture adds 0.12 in distribution and 0.23 out of distribution over no pretraining.
Dreaming raises task progress from 0.63 to 0.78 over seven settings. On Scoop it makes the trunk’s latents more alike across embodiments (mean Gromov-Wasserstein distance down 38.52%).
A coarse variant that merges camera views, pose streams and action outputs scores zero on simulated Kitchen-Cleanup after pretraining.

Limitations

The team collected all data themselves, so larger datasets are untested. Simulation covers one humanoid and two tasks. Adding a robot needs hand-specified modality correspondences and frames. Equal-sized batches reuse small datasets such as LocoMan’s more often, which may cause overfitting.

Videos

Out-of-distribution trial: the G1 inserts a larger brown block.

Out-of-distribution trial: the G1 inserts a white block.

Figures

From Figure 1: EgoX pretrains on egocentric demonstrations from humans and robots (top left), then fine-tunes on a ne... — From Figure 1: EgoX pretrains on egocentric demonstrations from humans and robots (top left), then fine-tunes on a new embodiment and task (bottom left). MXT+ routes each embodiment's observations through modality tokenizers into a shared Transformer trunk and decodes actions with embodiment-specific experts, while a training-only dreaming expert predicts future embodiment latents (center).

From Figure 3: Top: embodiment-specific tokenizers map images, proprioception and wrist, end-effector-link and torso ... — From Figure 3: Top: embodiment-specific tokenizers map images, proprioception and wrist, end-effector-link and torso poses into tokens that a shared encoder-decoder trunk fuses; action experts decode embodiment-specific action chunks, and dreaming experts predict future latents supervised by EMA copies of the tokenizers. Bottom: the embodiments, from Human-Dex5 to Z1-Tool, with their egocentric and wrist frames and end-effector keypoints.

From Figure 5: Normalized task progress of ACT, pi0.5, MXT+ without pretraining and MXT+ with cross-embodiment pretra... — From Figure 5: Normalized task progress of ACT, pi0.5, MXT+ without pretraining and MXT+ with cross-embodiment pretraining on 13 task-embodiment pairs (five physical robots and the simulated G1-Sim-Dex3), in and out of distribution.

Abstract

Generalist manipulation policies can benefit from the complementary experience of humans, real robots, and simulated agents. However, differences in morphology, sensing, and control interfaces make it difficult to learn effectively from their combined demonstrations. We introduce EgoX, an egocentric cross-embodiment learning framework that organizes heterogeneous demonstrations through semantically corresponding observation and action modalities. EgoX pretrains a Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) on this mixture and fine-tunes it on target-embodiment demonstrations. Within each embodiment, MXT+ assigns a dedicated tokenizer to each observation modality and a dedicated expert to each action modality, with a Transformer trunk shared across embodiments. A training-only embodiment-dreaming objective predicts latent representations of future visual and proprioceptive states, encouraging the shared trunk to model each embodiment's dynamics. Across 13 task-embodiment pairs spanning five tasks and six robot embodiments, including five physical robots and one simulated humanoid, cross-embodiment pretraining raises MXT+'s average normalized task progress from 0.63 to 0.80 over training from scratch. The gains are particularly pronounced under out-of-distribution conditions, where average task progress increases from 0.53 to 0.77. Ablations demonstrate the benefits of broader pretraining mixtures, modular interfaces, and embodiment dreaming, with dreaming producing more similar latent geometry across embodiments. Project page: egox-dream.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong