Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
TL;DR Trains Minitaur quadruped gaits with PPO in PyBullet and narrows the sim-to-real gap with a measured actuator model, simulated latency, randomization and a compact observation space. The learned gallop and trot run on the real robot without fine-tuning, as fast as expert-tuned gaits with 35% and 23% less power.
What they do
Policies learned in simulation tend to fail on real legged robots, because small model errors get amplified by frequent contact changes. The authors learn trotting and galloping for the Minitaur quadruped with PPO in PyBullet, deploy them with no fine-tuning on the robot, and measure which sim-to-real measures matter.
How it works
- Policy. The action is a user-given open-loop reference plus a learned, bounded feedback term, in leg space (swing and extension). A zero reference with wide bounds learns from scratch; a sine reference with ±0.25 rad bounds gives a trot. Reward is forward progress minus an energy penalty.
- Better simulator. Measured link masses, dimensions and motor friction; a DC-motor actuator model with piecewise-linear torque saturation in place of Bullet’s constraint-based position control; measured latency (3 ms for the PD servo, 15-19 ms for the policy on a Jetson TX2).
- Robust policies. Randomized mass, inertia, motor strength, friction, control step, latency, battery voltage and IMU noise; 130-220 N pushes on the base; a compact observation (4-D IMU only for trotting).
Results
- From scratch the policy learns to gallop: 1.34 m/s in simulation and 1.18 m/s on the robot. The trot reaches 0.50 m/s in simulation and 0.60 m/s on the robot.
- Against Ghost Robotics’ handcrafted gaits at similar speed, mechanical power drops 35% for galloping (188.79 vs 290.00 W) and 23% for trotting (71.78 vs 92.72 W).
- Controllers trained on the baseline simulator, even with perturbations, do poorly on the robot; dropping either the actuator model or latency breaks transfer.
- Randomization lowers peak return but makes it insensitive to changes such as inertia. A 12-D observation scores higher in simulation but worse on the robot; with randomization and the 4-D observation, all nine real runs trotted over 3 m without falling.
Limitations
Only straight running on flat ground: no speed or direction commands, no vision, no rough terrain. Randomization trades optimality for robustness, and real-robot results vary enough that each comparison used 100 trained controllers.
Videos
Figures
Abstract
Designing agile locomotion for quadruped robots often requires extensive expertise and tedious manual tuning. In this paper, we present a system to automate this process by leveraging deep reinforcement learning techniques. Our system can learn quadruped locomotion from scratch using simple reward signals. In addition, users can provide an open loop reference to guide the learning process when more control over the learned gait is needed. The control policies are learned in a physics simulator and then deployed on real robots. In robotics, policies trained in simulation often do not transfer to the real world. We narrow this reality gap by improving the physics simulator and learning robust policies. We improve the simulation using system identification, developing an accurate actuator model and simulating latency. We learn robust controllers by randomizing the physical environments, adding perturbations and designing a compact observation space. We evaluate our system on two agile locomotion gaits: trotting and galloping. After learning in simulation, a quadruped robot can successfully perform both gaits in the real world.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.