STRIDE: Learning Humanoid Locomotion in Minutes with Acceleration-Based Rewards
TL;DR STRIDE learns humanoid walking with a reward computed only from the onboard IMU, rewarding the robot for following a commanded acceleration profile instead of tracking base velocity. In MuJoCo simulation it learns to walk on the Unitree G1 and Booster T1 within 14 minutes of samples and cuts falls during training by 81%.
What they do
Learning to walk directly on a humanoid would avoid the sim-to-real gap. But standard locomotion rewards track base velocity, which needs motion capture or a well-calibrated state estimator, and humanoids fall often while learning. STRIDE uses a reward computed only from the onboard IMU and makes learning efficient enough that a single robot walks after minutes of samples. All experiments so far are in simulation.
How it works
- Acceleration reward. A reference velocity ramps toward the command with bounded acceleration. The robot integrates its IMU acceleration into a velocity estimate, and a PI law turns the gap to the reference into a goal acceleration. The reward tracks the low-pass-filtered goal acceleration. A “hold” term tracks the reference with the velocity estimate, because the acceleration is zero at constant speed.
- Observation. Joint states, the previous action, IMU acceleration, angular velocity and gravity direction, and the command, reference velocity and goal acceleration. No base velocity.
- Sample efficiency. Stiff PD joint control (gains 60 and 3), the off-policy XQC algorithm, and left-right mirroring of half of the replayed transitions.
Results
Setup: MuJoCo, 12 leg joints at 50 Hz, walking forward at 0.5 m/s, one robot, a budget of 83 minutes of samples.
- Every STRIDE run learns to walk after 12-14 minutes of samples on both robots. Final policies walk at 0.48 m/s and complete every episode.
- Without symmetry augmentation: 21 min (G1) and 18 min (T1). With low gains: 22 and 66 min. With neither, only half the runs learn within budget, taking 56 min on average.
- Average falls during training drop from over 2000 to 440. 99% of falls happen before a run learns to walk.
- A CPG gait prior did not change the time to walk. A single policy shared by both robots learned on the G1 but not on the T1.
Limitations
Simulation only, where the IMU acceleration has no noise or bias. On a real robot the IMU also measures gravity, and the authors expect the hold reward to accumulate its errors. Forward walking only, with at least two seeds per setting.
Figures
Abstract
Learning locomotion directly on a humanoid robot is a promising direction to avoid the sim-to-real gap. However, locomotion rewards in Reinforcement Learning (RL) typically track the base velocity, which cannot be measured onboard without an external motion capture system or well-calibrated state estimator. This is why we introduce STRIDE, which learns humanoid locomotion from minutes of samples with a reward that only requires the onboard Inertial Measurement Unit (IMU). Instead of tracking the base velocity, STRIDE rewards the robot for following a commanded acceleration profile. To learn efficiently from this reward, we use the off-policy RL algorithm XQC and combine a high gain setting for the joint position controllers with left-right symmetry augmentation. In simulation, STRIDE learns to walk on the Unitree G1 and Booster T1 humanoids within 14 minutes of samples. Compared to learning without these two components, it reduces the falls during training by 81%, while without them the robot learns only in over an hour.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.