Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning

Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H. Huang, Dhruva Tirumala, Jan Humplik, Markus Wulfmeier, Saran Tunyasuvunakool, Noah Y. Siegel, Roland Hafner, Michael Bloesch, Kristian Hartikainen, Arunkumar Byravan, Leonard Hasenclever, Yuval Tassa, Fereshteh Sadeghi, Nathan Batchelor, Federico Casarini, Stefano Saliceti, Charles Game, Neil Sreendra, Kushal Patel, Marlon Gwira, Andrea Huber, Nicole Hurley, Francesco Nori, Raia Hadsell, Nicolas Heess

Seminal April 2023

TL;DR A 20-joint Robotis OP3 miniature humanoid learns 1v1 soccer with deep RL in simulation and plays on real robots zero-shot. Compared with the robot's scripted controllers it walks 181% faster, turns 302% faster, gets up in 63% less time and kicks 34% faster.

What they do

A low-cost Robotis OP3 humanoid (51 cm, 3.5 kg, 20 servos) learns simplified 1v1 soccer with deep RL, trained only in MuJoCo and deployed zero-shot on real robots. One policy maps proprioception plus motion-capture game state (ball, opponent, goals) to joint position targets at 40 Hz.

How it works

Stage 1, skills. A soccer skill learns to score against an untrained opponent that just falls over. A get-up skill learns to reach target poses interpolated between three key poses taken from the scripted get-up controller. Both are trained with distributional MPO.
Stage 2, distillation and self-play. One agent is KL-regularized to the get-up skill when it is on the ground and to the soccer skill when it is upright. Each regularization weight drops to zero once predicted Q-values pass a threshold. Opponents come from the first quarter of the agent’s saved snapshots, and the critic is told which opponent it faces.
Sim-to-real. Actuator system identification, domain randomization (floor friction, joint offsets, IMU pose, up to 0.5 kg extra torso mass, 10-50 ms observation delay) and random pushes every 1-3 s. Shaping rewards penalize knee torque peaks and reward an upright torso.

Results

Against the scripted controllers on the real robot, it walks 181% faster, turns 302% faster and gets up in 63% less time. With a run-up, its kicks average 2.8 m/s (34% faster), with a maximum of 3.4 m/s.
In a get-up-and-shoot set piece it scored 29/50 on the real robot versus 35/50 in simulation, and it got up and kicked the ball in every real trial.
In a set piece with an opponent between it and the ball, it walked around the opponent in 10/10 trials and scored in 9.
Without skill regularization, agents either rolled to the ball or only learned to get up and stand still. Without randomization and pushes, real robots fell every one or two steps.

Limitations

The policy relies on motion capture, hand-chosen get-up poses, a manual choice of which skill to distill in each state, and tuned reward weights. The robots wore out quickly. The servos were not modeled, so behavior depended on battery charge, which limited play to 5-10 minutes per charge. Self-play was sometimes unstable.

Videos

Figures

From Figure 1: Matching simulated (left) and real (right) 5 m by 4 m soccer pitches; in the real one, motion capture ... — From Figure 1: Matching simulated (left) and real (right) 5 m by 4 m soccer pitches; in the real one, motion capture tracks both robots and the ball.

From Figure 2: Two-stage training. Stage 1 trains separate soccer and get-up skills; stage 2 distills both into one a... — From Figure 2: Two-stage training. Stage 1 trains separate soccer and get-up skills; stage 2 distills both into one agent that plays against opponents sampled from its own earlier snapshots.

From Figure 3: Behaviors of the trained policy on real robots: kicking a moving ball, blocking a shot, turning, getti... — From Figure 3: Behaviors of the trained policy on real robots: kicking a moving ball, blocking a shot, turning, getting up from the back and front, positioning between the ball and its own goal, and recovering from pushes.

From Figure 7: Win rate against six fixed opponents during training. The full method (self-play plus skill regulariza... — From Figure 7: Win rate against six fixed opponents during training. The full method (self-play plus skill regularization) beats training against the evaluation opponents directly, self-play against all snapshots, and training without skill regularization or reward shaping. Those last agents learn to roll to the ball instead of getting up (B).

Abstract

We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies in dynamic environments. We used Deep RL to train a humanoid robot with 20 actuated joints to play a simplified one-versus-one (1v1) soccer game. The resulting agent exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning, kicking and more; and it transitions between them in a smooth, stable, and efficient manner. The agent's locomotion and tactical behavior adapts to specific game contexts in a way that would be impractical to manually design. The agent also developed a basic strategic understanding of the game, and learned, for instance, to anticipate ball movements and to block opponent shots. Our agent was trained in simulation and transferred to real robots zero-shot. We found that a combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training in simulation enabled good-quality transfer. Although the robots are inherently fragile, basic regularization of the behavior during training led the robots to learn safe and effective movements while still performing in a dynamic and agile way -- well beyond what is intuitively expected from the robot. Indeed, in experiments, they walked 181% faster, turned 302% faster, took 63% less time to get up, and kicked a ball 34% faster than a scripted baseline, while efficiently combining the skills to achieve the longer term objectives.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong