RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On

Seungyong Lee, Suhyeok Jang, Minyoung Kim, Jinwoo Shin

November 2026

The video model fills in robot motion between generated keyframes, following the human hand skeleton.

TL;DR RoboTryOn turns egocentric human videos into robot demonstrations by generating a video of the robot making the same motion and reading its actions back with an inverse dynamics model. On two bimanual robots with dexterous hands, it lifts success on seven tasks that have no robot demonstrations from 2% to 60% on average.

What they do

Teaching a dexterous robot a new task usually means teleoperating it. RoboTryOn instead has a person do the task in front of a camera at the robot’s head-camera position, then converts that video into a robot demonstration the way virtual try-on swaps a garment: the motion, objects and background stay, and the human arm becomes the robot. A VLA (π0.5) is fine-tuned on these synthetic episodes, alone or mixed with teleoperated ones.

How it works

Keyframe translation. Keyframes are taken where the wrists slow down. A fine-tuned FLUX.2 image model inpaints the masked human arm with the robot, guided by a retargeted 2D hand skeleton and conditioned on the previous generated keyframe so the scene stays consistent.
Video synthesis. A LoRA on the LTX-2.5 video model fills in the motion between robot keyframes, following the human hand-skeleton flow.
Action labels. An inverse dynamics model on a Depth Anything V2 backbone reads joint actions from the generated video; inverse kinematics then corrects the wrist position to the reconstructed human wrist.
All three models train on robot data only (2.5-3.8 h per robot), with no paired human-robot data, and one set of models serves the OpenArm, RB-Y1 and GR-1.

Results

Two robots (OpenArm with 6-DoF Inspire hands, RB-Y1 with 20-DoF WujiHand2 hands), 11 tabletop tasks, 28 trials each. On the seven tasks with human videos only, success rises from 3.1% to 73.0% on the OpenArm and from 1.0% to 46.9% on the RB-Y1; EgoMimic and Masquerade stay below 10%.
On the four pick-and-place tasks that have teleoperated data, adding the synthetic episodes raises success from 47.3% to 75.9% (OpenArm) and 69.6% to 79.5% (RB-Y1).
Replayed open loop in simulation, the IDM labels succeed in 69.7% of episodes against 34.8% for retargeted hand poses.
Human videos alone match 50 teleoperated demonstrations per task at about a third of the collection person-hours.

Limitations

Each new robot still needs a few hours of teleoperated video to train the generative models and the IDM. Labels read from pixels carry no forces and are too imprecise for tasks like tight insertion. Human videos must come from the robot’s fixed head-camera viewpoint. No method completes Air Fryer on the RB-Y1, and generating one 15 s demonstration takes about 11.6 minutes on an H100.

Videos

Air Fryer, learned from human videos only: a human video (left) and the OpenArm rollout (right).

PnP Ball in Cup, a bimanual task learned from human videos only: human video (left), OpenArm rollout (right).

Figures

From Figure 1: RoboTryOn pipeline. (a) The image model translates keyframes of a human video into robot keyframes one... — From Figure 1: RoboTryOn pipeline. (a) The image model translates keyframes of a human video into robot keyframes one after another, the video model synthesizes the motion between them guided by the hand skeleton flow, and the inverse dynamics model labels the actions. (b) The image model inpaints the masked human frame, using the previous robot keyframe as a reference. (c) The video model synthesizes the robot video from the robot keyframes at their time positions and the skeleton flow.

From Figure 14: Long-horizon synthesis. A 25-second human video with its selected keyframes (top) and the generated r... — From Figure 14: Long-horizon synthesis. A 25-second human video with its selected keyframes (top) and the generated robot video (bottom), split into clips at the keyframe times. Adjacent clips share the keyframe at their boundary, so robot and objects stay consistent across clips.

From Figure 12: OpenArm rollouts on the seven tasks that had human videos only: a novel object, a novel goal, two nov... — From Figure 12: OpenArm rollouts on the seven tasks that had human videos only: a novel object, a novel goal, two novel skills (pouring, brushing) and three bimanual tasks.

Abstract

Robot foundation models have shown remarkable performance in robot control, yet adapting them to a new task still relies heavily on teleoperated demonstrations. Egocentric human videos offer a scalable substitute, but overlaying rendered robots on human videos or retargeting human poses into robot actions leaves significant visual or action gaps for high-DoF robot embodiments. To address this limitation, we propose RoboTryOn, which treats human-to-robot translation as virtual try-on: try-on keeps the person and swaps the garment, RoboTryOn keeps the demonstrated motion and swaps the human for the robot. Specifically, RoboTryOn converts a human video into a video of the target robot specified by a reference image, while preserving the original task, objects, and background. To maintain visual consistency, we translate selected keyframes autoregressively and use them to guide video synthesis with motion conditioned on the human hand skeleton. Additionally, we use an inverse dynamics model (IDM) that incorporates depth features to infer action labels from synthesized videos for policy learning. Our experiments across multiple humanoid embodiments show that policies trained with our synthetic data improve the average success rate on novel tasks, including challenging bimanual manipulation, from 2% to 60%, without any real robot demonstration of those tasks. Moreover, training with our synthetic data matches the performance of policies trained solely on teleoperated demonstrations at approximately a third of the collection cost, demonstrating the cost efficiency of our approach.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong