OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
TL;DR OmniH2O trains one whole-body humanoid policy that tracks only head and hand positions, so a Unitree H1 with dexterous hands can be driven by VR, an RGB camera, language or GPT-4o. It tracks 94.1% of 14k motions in simulation (H2O 87.5%), and diffusion policies trained on its 40 minutes of teleoperation data succeed 8/10 on average across four tasks.
What they do
OmniH2O trains a single whole-body policy for a Unitree H1 with Inspire hands that tracks a kinematic pose goal. Because many sources can produce that goal, the same policy serves VR teleoperation, an RGB camera, text-to-motion (MDM), GPT-4o picking motion primitives from head-camera images, and imitation-learning policies trained on data collected with it.
How it works
- Data. AMASS motions retargeted to H1, plus a “stable” copy of each with the root and lower body fixed standing or squatting, so the robot learns to stand still while its arms move.
- Teacher. A PPO policy with privileged full-body state. A “max feet height for each step” reward, with a curriculum, replaces stomping with standing still or taking large steps.
- Student. Distilled with DAgger. It sees only head and hand positions (the 3 points a VR headset gives) and 25 steps of joint and IMU history, with no global linear velocity, so it needs no motion capture. Hand joints come from inverse kinematics on VR hand poses.
- Autonomy. Image-conditioned policies predict head and hand goals plus hand commands; OmniH2O turns them into joint targets.
Results
- Simulation, 14k motions: 94.10% success, against 87.52% for H2O and 94.77% for the privileged teacher. Training the same student with RL instead of DAgger drops it to 47.11%. Tracking 3 points reaches 94.10%, all 22 points 94.72%.
- Real robot, 20 standing motions: global joint error 47.94 mm against 87.33 mm for H2O. Feeding in velocity from VIO or a learned estimator did worse than leaving it out.
- OmniH2O-6 dataset: about 40 minutes of VR demonstrations over six tasks, with head-camera RGBD at 30 Hz. Across four tasks, DP-DDPM succeeds 8/10 on average, DP-DDIM 7.75/10 and ResNet BC 1/10; with 25% of the data DP-DDPM falls to 4/10.
Limitations
Mapping operator poses into the robot frame needs root odometry, and VIO can be noisy or discontinuous, which throws off the motion goals. There are no safety guarantees for extreme disturbances or out-of-distribution goals such as large jumps in the command. Climbing stairs from sparse upper-body goals is left to future work.
Videos
Figures
Abstract
We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.