UniLatent: Cross-Embodiment Transfer via Latent Observation Alignment

Yufei Ding, Yi Du, Weiduo Yuan, Hao Jiang, Dylan Goetting, Lawrence Yunliang Chen, Daniel Seita, Yue Wang

November 2026

TL;DR UniLatent renders two robots at matched end-effector poses in simulation and trains a source image encoder to reproduce the features a frozen DINOv2 gives the target robot, so a policy trained on one arm's demonstrations runs on another. Zero-shot success in simulation averages 76.0% against 44.5% for pixel translation, and a real xArm-to-Franka transfer succeeds 27/30 with 10 target demonstrations per task.

What they do

Demonstrations collected on one robot arm transfer poorly to another, because a third-person camera sees a different arm, gripper and occlusion pattern even when the motion is the same. Prior work edits the images (masking, inpainting, diffusion) to look like the target robot. UniLatent closes the gap in feature space instead: it aligns a source-robot image encoder to the encoder the target robot will use, then trains the policy on those features.

How it works

Paired rendering. In IsaacLab/RoboVerse, both robots are placed at the same tool-center-point pose by inverse kinematics and rendered from the same camera in the same randomized scene. About 100k generic pairs, plus an optional 10k with the task’s objects.
Alignment. Both encoders start from DINOv2-base. The target encoder stays frozen; only the source encoder is trained, with an MSE loss, to output the features of the paired target image.
Latent policy. A flow-matching policy is trained on source demonstrations encoded by the aligned source encoder and predicts chunks of Cartesian end-effector poses with a binary gripper command.
Deployment. Target images go through the frozen target encoder. No image is translated at test time.

Results

Simulation, Franka to xArm7 and Sawyer, zero-shot, 100 rollouts per setting: direct transfer gets 0-5% despite 80-97% on the Franka. UniLatent averages 76.0% over six task-robot settings against 44.5% for RoVi-Aug*, a pixel-translation baseline built on the same aligned encoder.
The largest gain is on Stack Cup into Bowl: 15% to 65% on xArm and 5% to 51% on Sawyer.
Real xArm to Franka, fine-tuned on 10 Franka demonstrations per task: 27/30 rollouts succeed, against 23/30 with RoVi-Aug pretraining and 11/30 with target-only training.
Sampling manipulation-relevant poses for the pairs matters: 93%/81% on xArm/Sawyer against 60%/32% with unconstrained sampling.

Limitations

Only 7-DoF single arms with parallel-jaw grippers and a shared end-effector action interface are covered. Source and target share the task, scene and camera; use with external datasets such as Open X-Embodiment is not shown. The real-robot result is few-shot, not zero-shot, on a single physical setup, and robustness to real clutter and lighting is untested.

Figures

From Figure 1: UniLatent renders paired source-target images and aligns a source encoder to a target encoder, forming... — From Figure 1: UniLatent renders paired source-target images and aligns a source encoder to a target encoder, forming a shared latent space that reduces embodiment-induced visual gaps. Top: source and target embeddings increasingly overlap during training. Bottom: cross-decoded source observations become more target-consistent as alignment improves.

From Figure 2: Pipeline overview. Paired source and target observations are rendered for matched end-effector poses a... — From Figure 2: Pipeline overview. Paired source and target observations are rendered for matched end-effector poses and used to train the source encoder to match a fixed target encoder in a shared latent space. A latent policy is then trained on source demonstrations and deployed on the target robot by encoding target observations with the target encoder.

From Figure 5: Pixel-space translation examples. Each row shows a source observation, its paired target observation, ... — From Figure 5: Pixel-space translation examples. Each row shows a source observation, its paired target observation, the decoded source translation used by the RoVi-Aug* baseline, and the decoded target reconstruction.

From Figure 6: Real-world tasks (Close Drawer, Close Box Lid, Wipe Table). Each row shows frames from a teleoperated ... — From Figure 6: Real-world tasks (Close Drawer, Close Box Lid, Wipe Table). Each row shows frames from a teleoperated demonstration collected on the xArm under a fixed third-person camera.

Abstract

Robot datasets are costly to collect and are usually tied to a specific embodiment, making it difficult to reuse demonstrations across robots with different geometry, appearance, and kinematics. We introduce UniLatent, a cross-embodiment transfer framework that reduces the embodiment-induced observation gap in a shared latent space. UniLatent renders paired source-target robot observations in simulation and aligns a source visual encoder to a frozen target encoder. A policy trained on encoded source demonstrations can then run directly on target observations, with no pixel-level image translation. Across three simulation tasks and two target robot arms, UniLatent reaches up to 93% zero-shot success, exceeding the strongest pixel-translation baseline by over 30 points on average. In real-world xArm-to-Franka transfer, fine-tuning with only 10 target demonstrations per task yields 27/30 successful rollouts. Code, videos and data are available at https://selina2023.github.io/unilatent/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong