Expressive Whole-Body Control for Humanoid Robots

Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, Xiaolong Wang

Seminal February 2024

The robot adjusts its steps to keep up with fast arm swings copied from a human reference.

TL;DR One RL policy for the Unitree H1 humanoid imitates the upper body of about 780 CMU MoCap clips while its legs only track the clip's root velocity and orientation. This beats full-body imitation on velocity tracking in simulation (reward 318.7 vs 246.1) and runs on the real robot, which dances, high-fives and shakes hands with people.

What they do

Human motion-capture data has far more degrees of freedom than a real humanoid (69 in physics-based animation vs 19 on a Unitree H1), and animation controllers use actuator gains about ten times higher than hardware allows. ExBody therefore does not copy the reference exactly. One goal-conditioned policy imitates the upper body of a human clip, while the legs only have to follow the clip’s root velocity, orientation and height robustly. It is trained in Isaac Gym and deployed on the real H1.

How it works

Data. About 780 CMU MoCap clips (3.7 h) of single-person motion on flat ground, mostly walking plus dance, basketball and punching. Joint rotations are mapped directly onto the H1, treating its three-motor shoulders and hips as spherical joints.
Goals. The expression goal is 9 upper-body joint angles plus 3D keypoints of shoulders, elbows and hands. The root goal is linear velocity, roll/pitch/yaw and height. The policy sees only proprioception and a yaw error, not global velocity or height, and outputs PD targets for all 19 joints.
Training. PPO with imitation rewards for the upper body, tracking rewards for the root and regularizers for the feet and smoothness. Each environment starts from a state sampled from the dataset (reference state initialization, RSI) on randomized terrain.

Results

In simulation, ExBody has the best velocity tracking reward: 318.7, vs 246.1 for full-body tracking (worse on every metric), 205.6 with an added AMP reward and 181.9 with randomly sampled root goals, where the policy learns to kneel. Without RSI, mean episode length drops from 16.87 to 0.23.
On the real robot, mean absolute roll/pitch over 14 motions is 0.051 for ExBody vs 0.087 with AMP, whose straight-legged gait stumbles. Motions include dancing, hugging, handshakes and clips generated from text by a diffusion model (MDM).
Although trained on clips, the policy also follows randomly sampled velocity commands, covering a wider range than the training data.

Limitations

Direct joint-angle retargeting to a robot with fewer DoF loses information, so replayed motion can deviate from the original. Falls can break parts of the heavy, expensive robot, and there is no protection or recovery system. The legs do not imitate the reference at all.

Videos

Dancing with a person outdoors.

Punching a bag indoors, then interacting with a person outside.

Figures

From Figure 2: Overview. Motion clips from human motion datasets, generative models or video-to-pose models are retar... — From Figure 2: Overview. Motion clips from human motion datasets, generative models or video-to-pose models are retargeted to the robot. The policy is conditioned on an expression goal and a root movement goal extracted from each clip; the root goal can also come from a joystick.

From Figure 4: Left, human motion clips and their retargeted versions used for training. Right, at deployment the rob... — From Figure 4: Left, human motion clips and their retargeted versions used for training. Right, at deployment the robot replays motions from datasets, diffusion models or video-to-skeleton models. The H1 has 19 DoF; each shoulder and hip has three perpendicular joints.

From Figure 7: A high five on the real robot. Top, ExBody walks with bent knees and more foot clearance. Bottom, ExBo... — From Figure 7: A high five on the real robot. Top, ExBody walks with bent knees and more foot clearance. Bottom, ExBody + AMP walks straight-legged with less foot clearance.

Abstract

Can we enable humanoid robots to generate rich, diverse, and expressive motions in the real world? We propose to learn a whole-body control policy on a human-sized robot to mimic human motions as realistic as possible. To train such a policy, we leverage the large-scale human motion capture data from the graphics community in a Reinforcement Learning framework. However, directly performing imitation learning with the motion capture dataset would not work on the real humanoid robot, given the large gap in degrees of freedom and physical capabilities. Our method Expressive Whole-Body Control (Exbody) tackles this problem by encouraging the upper humanoid body to imitate a reference motion, while relaxing the imitation constraint on its two legs and only requiring them to follow a given velocity robustly. With training in simulation and Sim2Real transfer, our policy can control a humanoid robot to walk in different styles, shake hands with humans, and even dance with a human in the real world. We conduct extensive studies and comparisons on diverse motions in both simulation and the real world to show the effectiveness of our approach.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong