DTC: Deep Tracking Control

Fabian Jenelten, Junzhe He, Farbod Farshidian, Marco Hutter

Seminal September 2023

TL;DR DTC trains an RL policy in simulation to track footholds from a trajectory optimizer, giving the ANYmal quadruped model-based foot-placement accuracy together with RL robustness. It tracks footholds to 2.3 cm and crosses gaps, beams and stepping stones where an RL baseline fails.

What they do

Trajectory optimization (TO) places feet precisely but breaks when its model assumptions fail; RL is robust but struggles where valid footholds are rare, such as gaps and stepping stones. DTC has a TO planner (TAMOLS) propose footholds and trains an RL policy in simulation to track them on ANYmal.

How it works

Observations. Planar footholds, desired joint positions at touch-down, contact state and a height scan sampled from each foot to its foothold. The base-pose reference goes only to the critic, which made the policy less sensitive to map errors and independent of the planner.
Rewards. Soft exponential base-trajectory tracking, a hard logarithmic foothold reward given once per step at touch-down, and a consistency reward between successive plans.
Training. 4096 robots on 12 terrain types for two weeks. The optimizer re-solves at each touch-down rather than every policy step, which avoids a “lazy” local optimum and cuts compute 23x. Height-scan noise and per-foot drift keep the policy from over-relying on perception.
Deployment. Policy at 50 Hz; the optimizer runs at about 400 Hz in a separate thread.

Results

Footholds are tracked to 2.3 cm on average on flat ground.
An RL baseline failed a 0.1 m gap and a 1.0 m beam. DTC crossed the gap 10/10 times, four 0.6 m gaps and a 1.8 m beam 4/4 times each, and 0.2 × 0.2 m stepping stones 10/10 times.
On a floor with a hidden gap, the TAMOLS-based controller failed while DTC got across; over slippery and deformable obstacles it did not fall in 5/5 trials.
Paired zero-shot with a different MPC planner, it climbed a 0.48 m box, 50% higher than the RL baseline. In simulation it beat an RL policy retrained on the same terrains on sparse terrains.

Limitations

Two weeks of training, no more sample-efficient than comparable RL approaches. The gait is a fixed trot. With TAMOLS’s simplified kinematics, knee collisions limit climbing to about 0.40 m.

Figures

From Figure 1: ANYmal walking indoors and outdoors with DTC, on terrain where steppable surfaces are sparse (bottom l... — From Figure 1: ANYmal walking indoors and outdoors with DTC, on terrain where steppable surfaces are sparse (bottom left) and where environmental uncertainty is high (top right).

From Figure 8: (A) The optimizer gives footholds, base pose, twist and acceleration, and desired joint positions; a h... — From Figure 8: (A) The optimizer gives footholds, base pose, twist and acceleration, and desired joint positions; a height scan is sampled between each foot and its foothold. (B) Training: the optimizer re-solves at each touch-down and the policy is rewarded for driving the robot toward the solution; actor observations are noisy, while the critic and optimizer get ground truth. (C) Deployment: the policy turns optimized footholds into target joint positions for a PD controller. (D) The 12 training terrain types.

From Figure 5: (A) The RL baseline gets stuck when its hind legs fall into a small gap and (B) stumbles along a narro... — From Figure 5: (A) The RL baseline gets stuck when its hind legs fall into a small gap and (B) stumbles along a narrow beam. With DTC, the robot crosses (C) four consecutive large gaps and (D) a long beam.

From Figure 7: Simulation results. (A) Success and failure rates for optimizer update rates up to 50 Hz. (B) Comparis... — From Figure 7: Simulation results. (A) Success and failure rates for optimizer update rates up to 50 Hz. (B) Comparison with RL baselines on all 120 terrains and on terrains with dense (white) and sparse (gray) footholds. (C) Tracking error and success rate under elevation-map drift. (D) Terrain level and foothold reward during training, with and without desired joint positions.

Abstract

Legged locomotion is a complex control problem that requires both accuracy and robustness to cope with real-world challenges. Legged systems have traditionally been controlled using trajectory optimization with inverse dynamics. Such hierarchical model-based methods are appealing due to intuitive cost function tuning, accurate planning, generalization, and most importantly, the insightful understanding gained from more than one decade of extensive research. However, model mismatch and violation of assumptions are common sources of faulty operation. Simulation-based reinforcement learning, on the other hand, results in locomotion policies with unprecedented robustness and recovery skills. Yet, all learning algorithms struggle with sparse rewards emerging from environments where valid footholds are rare, such as gaps or stepping stones. In this work, we propose a hybrid control architecture that combines the advantages of both worlds to simultaneously achieve greater robustness, foot-placement accuracy, and terrain generalization. Our approach utilizes a model-based planner to roll out a reference motion during training. A deep neural network policy is trained in simulation, aiming to track the optimized footholds. We evaluate the accuracy of our locomotion pipeline on sparse terrains, where pure data-driven methods are prone to fail. Furthermore, we demonstrate superior robustness in the presence of slippery or deformable ground when compared to model-based counterparts. Finally, we show that our proposed tracking controller generalizes across different trajectory optimization methods not seen during training. In conclusion, our work unites the predictive capabilities and optimality guarantees of online planning with the inherent robustness attributed to offline learning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong