HumanPlus: Humanoid Shadowing and Imitation from Humans

Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, Chelsea Finn

Seminal June 2024

The humanoid autonomously folds a sweatshirt on a table (2x speed).

TL;DR A humanoid learns to copy a person's body and hand motion live from one RGB camera, using an RL policy trained on 40 hours of human motion data; the copied demonstrations then train vision-based skill policies. With up to 40 demonstrations, a 33-DoF humanoid completes six tasks, from folding a sweatshirt to putting on a shoe and walking, at 60-100% success.

What they do

HumanPlus is a full-stack system for a humanoid to learn from human data. An RL policy trained on human motion lets the robot shadow an operator in real time from a single RGB camera. The shadowing doubles as whole-body teleoperation, and the collected demonstrations train autonomous skill policies from the robot’s head cameras.

How it works

Hardware. A Unitree H1 extended to 33 DoF: two 6-DoF Inspire hands, two custom 1-DoF wrists, a 19-DoF body, and two head-mounted RGB webcams.
Shadowing policy. AMASS (40 hours, over 11,000 sequences) is retargeted to the humanoid by copying joint angles. A decoder-only transformer, trained with PPO and domain randomization, maps proprioception and a target pose to 19 joint setpoints at 50 Hz and transfers zero-shot. WHAM and HaMeR estimate body and hand poses live.
Imitation policy. The Humanoid Imitation Transformer, a decoder-only ACT variant, takes binocular image features and proprioception and predicts 50 target poses. It also predicts future image features under an L2 loss, which keeps it from overfitting to proprioception.

Results

Teleoperation user study (6 participants): Rearrange Objects takes 5.20 s versus 6.60-8.87 s for kinesthetic teaching, ALOHA and Meta Quest, with one operator and a $50 camera. Only HumanPlus could do the squatting variant.
Robustness: withstands 32-100 N pushes versus 24-40 N for the default H1 controller, and recovers in 1.2 s versus 15 s.
Imitation: HIT succeeds 100% on Fold Clothes, 90% on Rearrange, Greeting and Warehouse, 80% on Type “AI”, and 60% on Wear a Shoe and Walk, where monocular HIT, ACT and open-loop replay all score 0%.

Limitations

The hardware has fewer DoF than a human (1-DoF ankles, 5-DoF arms), and the fixed head cameras can lose sight of the hands. Retargeting is a fixed mapping that drops joints, pose estimation fails under heavy occlusion, and long-horizon navigation is not addressed.

Videos

The humanoid grasps a spray can from a warehouse shelf, squats and places it on a cart.

The humanoid autonomously picks up soft toys and places them in a basket.

Figures

From Figure 2: Shadowing pipeline. One RGB camera gives body and hand pose estimates, which are retargeted to humanoi... — From Figure 2: Shadowing pipeline. One RGB camera gives body and hand pose estimates, which are retargeted to humanoid poses and tracked by the Humanoid Shadowing Transformer.

From Figure 3: The two decoder-only transformers: the Humanoid Shadowing Transformer for low-level control and the Hu... — From Figure 3: The two decoder-only transformers: the Humanoid Shadowing Transformer for low-level control and the Humanoid Imitation Transformer for imitation learning.

From Figure 6: Maximum push forces the robot withstands in each direction. The learned low-level policy tolerates lar... — From Figure 6: Maximum push forces the robot withstands in each direction. The learned low-level policy tolerates larger forces than the H1 default controller.

Abstract

One of the key arguments for building robots that have similar form factors to human beings is that we can leverage the massive human data for training. Yet, doing so has remained challenging in practice due to the complexities in humanoid perception and control, lingering physical gaps between humanoids and humans in morphologies and actuation, and lack of a data pipeline for humanoids to learn autonomous skills from egocentric vision. In this paper, we introduce a full-stack system for humanoids to learn motion and autonomous skills from human data. We first train a low-level policy in simulation via reinforcement learning using existing 40-hour human motion datasets. This policy transfers to the real world and allows humanoid robots to follow human body and hand motion in real time using only a RGB camera, i.e. shadowing. Through shadowing, human operators can teleoperate humanoids to collect whole-body data for learning different tasks in the real world. Using the data collected, we then perform supervised behavior cloning to train skill policies using egocentric vision, allowing humanoids to complete different tasks autonomously by imitating human skills. We demonstrate the system on our customized 33-DoF 180cm humanoid, autonomously completing tasks such as wearing a shoe to stand up and walk, unloading objects from warehouse racks, folding a sweatshirt, rearranging objects, typing, and greeting another robot with 60-100% success rates using up to 40 demonstrations. Project website: https://humanoid-ai.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong