TWIST: Teleoperated Whole-Body Imitation System

Yanjie Ze, Zixuan Chen, João Pedro Araújo, Zi-ang Cao, Xue Bin Peng, Jiajun Wu, C. Karen Liu

Seminal May 2025

TL;DR A MoCap system streams an operator's whole-body motion to a Unitree G1, and one tracking policy trained with RL plus behavior cloning turns it into balanced skills like crouching to lift a box, kicking a ball and dancing. Adding 150 in-house MoCap clips to training cuts summed tracking error on unseen motions from 2.09 to 1.52.

What they do

TWIST teleoperates a humanoid by having it imitate the operator’s whole-body motion in real time. OptiTrack MoCap captures the human, an online retargeter converts the pose into robot joint positions and root velocities, and a single controller tracks them while keeping balance. It runs on a 29-DoF, 1.3 m Unitree G1.

How it works

Data. Over 15,000 AMASS and OMOMO clips (~42 h) retargeted offline, plus 150 in-house MoCap clips (~0.5 h) retargeted online, so training sees real teleoperation noise. The online retargeter jointly optimizes 3D joint positions and orientations.
Teacher. PPO policy that sees 2 s of future reference frames and tracks targets in the robot’s local frame, with penalties for jitter and foot slip.
Student. Sees only proprioception and the current frame; trained with PPO plus a decaying KL term toward the teacher (RL+BC). Large end-effector perturbations in training teach it to exert force.
Deployment. Zero-shot sim-to-real: MoCap at 120 Hz, retargeting and policy at 50 Hz, PD control at 1000 Hz.

Results

One controller handles uprighting a trash can, lifting a box from the floor, kicking a door shut while carrying objects, kicking a soccer ball, crouching under obstacles, boxing and waltzing.
On 50 unseen MoCap clips, RL+BC beats pure RL on all six tracking metrics (joint position 0.083 vs 0.090 rad). Feet error is 23.4 mm, vs 26.5 for RL, 33.7 without in-house MoCap data and 35.1 for DAgger.
In-house clips cut summed tracking error from 2.09 to 1.52 with the improved retargeter (2.36 to 2.09 with the original).
Total delay is about 0.9 s: 0.7 s to generate tracking goals, 0.2 s for the policy.

Limitations

No egocentric video or touch feedback reaches the operator, so occlusions and grasp success are hard to judge. The G1’s motors overheat after 5-10 minutes, especially when crouching, causing most failures. The MoCap setup is not portable.

Figures

From Figure 1: TWIST teleoperates a humanoid in real time from whole-body human motion, using a single neural network... — From Figure 1: TWIST teleoperates a humanoid in real time from whole-body human motion, using a single neural network controller.

From Figure 3: The three stages: retargeting Internet and in-house MoCap data into a humanoid motion dataset, trainin... — From Figure 3: The three stages: retargeting Internet and in-house MoCap data into a humanoid motion dataset, training one whole-body controller in simulation, and teleoperating the real robot with MoCap.

From Figure 2: Real-robot skills: whole-body manipulation (lifting boxes from the ground), legged manipulation (kicki... — From Figure 2: Real-robot skills: whole-body manipulation (lifting boxes from the ground), legged manipulation (kicking a football), locomotion (walking sideways) and expressive motion (Waltz dance).

From Figure 6: Tracking errors of RL, DAgger and RL+BC controllers on MoCap test data (left), and their behaviors (ri... — From Figure 6: Tracking errors of RL, DAgger and RL+BC controllers on MoCap test data (left), and their behaviors (right): pure RL slides its feet, DAgger sometimes loses balance.

Abstract

Teleoperating humanoid robots in a whole-body manner marks a fundamental step toward developing general-purpose robotic intelligence, with human motion providing an ideal interface for controlling all degrees of freedom. Yet, most current humanoid teleoperation systems fall short of enabling coordinated whole-body behavior, typically limiting themselves to isolated locomotion or manipulation tasks. We present the Teleoperated Whole-Body Imitation System (TWIST), a system for humanoid teleoperation through whole-body motion imitation. We first generate reference motion clips by retargeting human motion capture data to the humanoid robot. We then develop a robust, adaptive, and responsive whole-body controller using a combination of reinforcement learning and behavior cloning (RL+BC). Through systematic analysis, we demonstrate how incorporating privileged future motion frames and real-world motion capture (MoCap) data improves tracking accuracy. TWIST enables real-world humanoid robots to achieve unprecedented, versatile, and coordinated whole-body motor skills--spanning whole-body manipulation, legged manipulation, locomotion, and expressive movement--using a single unified neural network controller. Our project website: https://humanoid-teleop.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong