Perpetual Humanoid Control for Real-time Simulated Avatars

Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, Weipeng Xu

Seminal May 2023

TL;DR A single physics-based humanoid controller (PHC) imitates 98.9% of the 11,313 AMASS training clips without any external stabilizing force, and gets up after falls to resume tracking. It drives avatars from noisy video pose estimates (about 90% success on H36M) or language-generated motion without resets.

What they do

Simulated humanoids driven by noisy pose estimates tend to fall. Prior imitators either reset to the (unreliable) reference pose or push the root with a non-physical residual force. PHC is a single RL policy that tracks the AMASS MoCap dataset with no external force, gets up after a fall, walks back to the reference and resumes imitation. It accepts full poses or 3D keypoints only.

How it works

Imitation. SMPL humanoid with 23 actuated joints in Isaac Gym, trained with PPO. Actions are PD targets, not residuals on the reference pose. Reward is tracking plus an AMP discriminator term plus an energy penalty; early termination ignores ankles and toes.
Progressive primitives. Train a primitive on all clips, collect the clips it fails, freeze it, and train a new one (warm-started from its weights) on that harder subset.
Fail-state recovery. A last primitive learns to get up from random falls and reach a reference 2-5 m away; its goal keeps only the reference root, and it trains on walking and running clips.
Composer. A multiplicative control policy weights all four frozen primitives at every step.

Results

MoCap: 98.9% success on AMASS train (UHC with residual force 97.0%, without 84.5%); 96.4% on AMASS test and 92.9% on unseen H36M motion.
Noisy H36M video poses: 88.7% success, 91.9% for the keypoint version, against 58.1% for UHC with residual force.
Ablation: one primitive reaches 59.4%, a composer without progressive training 66.2%, progressive training 86.9%; adding recovery does not hurt (88.7%).
Recovery within 10 s in 98.8% (fallen), 99.5% (3 m away) and 98.8% (both) of 1000 trials. Drives avatars live from a webcam at 30 fps.

Limitations

Highly dynamic clips (back flips, cartwheels, high jumps) still fail. Training takes about a week on one A100. Live use suffers from depth jitter, fluctuating frame rate and identity switches, assumes a level camera, and recovery gait shows artifacts and a jolt when imitation resumes.

Figures

From Figure 1: Left, a real-time avatar driven by webcam video, where the blue humanoid recovers from a fall. Right, imitating three disjoint language-generated motion clips, with the controller filling in between them; the color gradient shows time.

From Figure 3: The goal-conditioned RL framework with an adversarial motion prior. A composer weights frozen pretrain... — From Figure 3: The goal-conditioned RL framework with an adversarial motion prior. A composer weights frozen pretrained primitives to produce the action in Isaac Gym; training uses PPO with a goal reward plus a discriminator reward.

From Figure 2: Progressive training. Each new primitive learns the harder sequences the previous ones failed on; a fa... — From Figure 2: Progressive training. Each new primitive learns the harder sequences the previous ones failed on; a fail-recovery primitive is trained last on simple locomotion data, then a composer combines the frozen primitives.

From Figure 4: Imitating MoCap (spin and kick), recovering from a fall back to the reference motion (red dots), imita... — From Figure 4: Imitating MoCap (spin and kick), recovering from a fall back to the reference motion (red dots), imitating noisy pose estimates from video and motion generated from language, and a real-time avatar driven by a webcam stream.

Abstract

We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e.g. pose estimates from video or generated from language) and unexpected falls. Our controller scales up to learning ten thousand motion clips without using any external stabilizing forces and learns to naturally recover from fail-state. Given reference motion, our controller can perpetually control simulated avatars without requiring resets. At its core, we propose the progressive multiplicative control policy (PMCP), which dynamically allocates new network capacity to learn harder and harder motion sequences. PMCP allows efficient scaling for learning from large-scale motion databases and adding new tasks, such as fail-state recovery, without catastrophic forgetting. We demonstrate the effectiveness of our controller by using it to imitate noisy poses from video-based pose estimators and language-based motion generators in a live and real-time multi-person avatar use case.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong