Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion

Gwanghyeon Ji, Juhyeok Mun, Hyeongjun Kim, Jemin Hwangbo

Seminal February 2022

TL;DR Train a quadruped's locomotion policy and a neural state estimator together in one PPO loop, with the estimator supplying base velocity, foot height and contact probability to the policy. On a real Mini Cheetah it runs at up to 3.75 m/s on flat ground and 3.54 m/s on a slippery plate with friction 0.22.

What they do

RL locomotion controllers usually need an accurate state estimate, but analytical estimators become unreliable on ice or sand, and many assume a fixed gait schedule that learned policies don’t follow. This paper replaces the estimator with a small network trained at the same time as the policy. The estimator outputs interpretable states (base linear velocity, foot height, contact probability) and feeds them to the policy, using only IMU and joint encoders. Everything is trained in RaiSim and transferred to a Mini Cheetah.

How it works

Concurrent training. After each batch of simulated trajectories, the actor and critic are updated with PPO and the estimator with supervised regression to ground truth, so the policy learns to cope with the estimator’s errors, e.g. under foot slip.
Networks. All three are MLPs; the inputs include joint-state history at 0.02, 0.04 and 0.06 s back. The policy outputs joint position targets for a PD controller, with policy and estimator both running at 100 Hz.
Foot height estimate. A reward term alone did not keep enough foot clearance at low speeds, so the estimated foot height is given directly to the policy.
Sim-to-real. A forward-velocity curriculum (up to 3.5 m/s) plus randomized observation noise, motor friction, PD gains, foot geometry and ground friction (0.4-1.0).

Results

Simulation: lowest velocity tracking error (Vx RMS 0.111 m/s, vs 0.168 implicit and 0.136 sequential). Average time to fall on rough terrain is 85.7 s, vs 25.0 s without the estimator and 30.0 s for implicit.
Real-robot top speed: 3.75 m/s on normal ground and 3.54 m/s at friction 0.22, vs 1.72 / 1.34 m/s for the built-in MPC.
The slope-trained model climbs a 19.1° hill (trained on ±10°) and a slippery 9.0° slope; the other controllers manage at most 12.4°.
The estimator forward pass takes 7 µs, against 34 µs for the robot’s linear Kalman filter.

Limitations

Blind, proprioceptive locomotion only. On hardware, estimated contacts lag true contacts by 0.04 s, which the authors attribute to chain and rubber-foot compliance and sparse joint history. Foot height is inferred assuming even terrain. The slope-trained model has slightly higher tracking error than the flat-trained one, which the authors attribute to the latter being fit to simpler terrain.

Figures

From Figure 1: The Mini Cheetah running across a slippery plate (friction coefficient 0.22) at an average speed of 3.... — From Figure 1: The Mini Cheetah running across a slippery plate (friction coefficient 0.22) at an average speed of 3.54 m/s.

From Figure 2: The estimator network maps sensor observations to base linear velocity, foot height and contact probab... — From Figure 2: The estimator network maps sensor observations to base linear velocity, foot height and contact probabilities, which are fed to the actor together with the observations. The estimator is trained with supervised MSE against simulator ground truth; the actor (outputting desired joint positions) and critic are trained with PPO.

From Figure 4: Estimated contact probability for the front right foot on the real robot, plotted against true contact... — From Figure 4: Estimated contact probability for the front right foot on the real robot, plotted against true contact, knee joint position error and joint velocity. Contact starts with an abrupt change in joint velocity at impact and ends with upward joint motion.

Abstract

In this paper, we propose a locomotion training framework where a control policy and a state estimator are trained concurrently. The framework consists of a policy network which outputs the desired joint positions and a state estimation network which outputs estimates of the robot's states such as the base linear velocity, foot height, and contact probability. We exploit a fast simulation environment to train the networks and the trained networks are transferred to the real robot. The trained policy and state estimator are capable of traversing diverse terrains such as a hill, slippery plate, and bumpy road. We also demonstrate that the learned policy can run at up to 3.75 m/s on normal flat ground and 3.54 m/s on a slippery plate with the coefficient of friction of 0.22.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong