Humanoid Parkour Learning

Ziwen Zhuang, Shenzhe Yao, Hang Zhao

Seminal June 2024

H1 clears a low hurdle and a tilted ramp, then jumps onto a platform; insets show the onboard depth image.

TL;DR One depth-vision RL policy lets a Unitree H1 humanoid jump, leap, climb stairs and clear hurdles, trained on fractal-noise terrain instead of motion references or foot-raising rewards. On the real robot it jumps onto 0.42 m platforms, leaps 0.8 m gaps and runs at 1.8 m/s.

What they do

Learning-based humanoid locomotion has mostly stayed on flat ground and relied on motion references or rewards that force the feet to lift. This paper trains one end-to-end policy for the Unitree H1 that uses a head-mounted depth camera to jump up and down, leap gaps, climb stairs and clear hurdles with no motion prior, and transfers it zero-shot to the real robot.

How it works

Fractal-noise walking. A plane-walking policy is trained with PPO in Isaac Gym on terrain with fractal noise in the height field, which makes the robot lift its feet without a feet-air-time reward. Random initial headings and an automatic yaw command preserve turning.
Oracle parkour policy. Starting from the walking policy, a GRU-MLP actor reads 11x19 terrain scandots plus proprioception and trains on 10 obstacle types with an auto-curriculum. Extra rewards penalize penetrating virtual obstacles and reward stepping near stair centers.
Depth distillation. A CNN on 48x64 depth images replaces the scandot encoder; everything else starts from the oracle. DAgger with an L1 action loss runs on four Nvidia 3090 GPUs, with simulated RealSense D435i noise.
Deployment. On the onboard i7 CPU, the depth encoder runs at 10 Hz and the policy at 50 Hz.

Results

Real robot: jumps onto a 0.42 m platform, leaps 0.8 m gaps, runs at 1.8 m/s. On jump, leap, stairs and hurdle tests (10 trials each) it beats a blind walking policy and the H1’s built-in MPC controller.
Simulation, fractal noise vs feet-air-time reward: 90 vs 80% success on jump up, 80 vs 50% on leap, 100 vs 20% on hurdle, 100 vs 30% on wave terrain.
After 24 h of distillation: 85/80/100/95% on jump/leap/stairs/hurdle, vs 40/45/65/25% with one GPU (4.1M vs 432M transitions) and at most 10% from a randomly initialized student.
With the arms teleoperated over the policy’s commands, the robot keeps its balance.

Limitations

The policy relies on manually constructed terrain and is hard to generalize to unseen terrain without further training. Harder manipulation skills would need further tuning to avoid disrupting the visual system.

Videos

Jumping onto a box, leaping a gap to the next box and jumping down.

Walking up small stairs built from boxes.

Figures

From Figure 1: A single vision-based end-to-end whole-body-control parkour policy lets the humanoid jump onto 0.42 m ... — From Figure 1: A single vision-based end-to-end whole-body-control parkour policy lets the humanoid jump onto 0.42 m platforms, leap over hurdles and 0.8 m gaps, and cross various terrains.

From Figure 3: Points are bound to each rigid body of the robot. Green points do not penetrate the virtual obstacle; ... — From Figure 3: Points are bound to each rigid body of the robot. Green points do not penetrate the virtual obstacle; red points do, at penetration depth d(p).

From Figure 5: Real-world success rates for jump height, leap length, stairs height and hurdle height, 10 trials per ... — From Figure 5: Real-world success rates for jump height, leap length, stairs height and hurdle height, 10 trials per setting, comparing the parkour policy with a blind walking policy and the H1's built-in MPC controller.

Abstract

Parkour is a grand challenge for legged locomotion, even for quadruped robots, requiring active perception and various maneuvers to overcome multiple challenging obstacles. Existing methods for humanoid locomotion either optimize a trajectory for a single parkour track or train a reinforcement learning policy only to walk with a significant amount of motion references. In this work, we propose a framework for learning an end-to-end vision-based whole-body-control parkour policy for humanoid robots that overcomes multiple parkour skills without any motion prior. Using the parkour policy, the humanoid robot can jump on a 0.42m platform, leap over hurdles, 0.8m gaps, and much more. It can also run at 1.8m/s in the wild and walk robustly on different terrains. We test our policy in indoor and outdoor environments to demonstrate that it can autonomously select parkour skills while following the rotation command of the joystick. We override the arm actions and show that this framework can easily transfer to humanoid mobile manipulation tasks. Videos can be found at https://humanoid4parkour.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong