RMA: Rapid Motor Adaptation for Legged Robots

Ashish Kumar, Zipeng Fu, Deepak Pathak, Jitendra Malik

★ Must read July 2021

The A1 walking across loose rocks on a shoreline.

TL;DR RMA trains a quadruped walking policy and an online adaptation module entirely in simulation; the module infers a latent environment code from the last 0.5 s of joint states and actions, so a Unitree A1 adapts in fractions of a second with no real-world fine-tuning. It carries 12 kg, its own body weight, and reaches 73.5% success in simulation versus 62.4% for domain randomization.

What they do

Simulation-trained walking policies break when real terrain, payload or motors differ from the simulator, and collecting real data to adapt risks falls. RMA trains, entirely in simulation, a walking policy plus a module that infers the environment from the robot’s recent motion, so a Unitree A1 adapts in fractions of a second with no real-world fine-tuning or reference gaits.

How it works

Base policy. An MLP maps proprioceptive state, the previous action and an 8-dim latent “extrinsics” vector to 12 target joint angles. An encoder compresses 17 privileged parameters (payload, motor strength, friction, terrain height) into the extrinsics; both train jointly with PPO on fractal terrain in RaiSim.
Natural rewards. Bioenergetics-inspired terms penalize work, ground impact, jerk and foot slip, ramped up by a curriculum so the robot does not learn to stand still.
Adaptation module. A 1-D CNN regresses the extrinsics from the last 0.5 s of states and actions, trained by supervised learning on on-policy rollouts.
Deployment. The module runs at 10 Hz and the base policy at 100 Hz, asynchronously, on board.

Results

Simulation: 73.5% success, close to an expert given true extrinsics (76.2%), versus 62.4% for domain randomization, 56.5% for system identification and 52.1% without adaptation.
Indoors: carries up to 12 kg (its body weight) and steps down 15 cm with 80% success; A1’s built-in controller fails on uneven foam and degrades from 8 kg of payload.
Outdoors: no failures on sand, mud and dirt or in tall vegetation, 70% down trail stairs never seen in training, 80% across cement and pebble piles, 90% across an oily patch.

Limitations

Proprioception only: sudden drops on stairs or several legs caught on rocks sometimes cause falls, and the authors name vision as the next step. Real-world comparisons use two baselines and 5 trials to spare the hardware.

Videos

Walking down a dirt pile beside a road.

Crossing a pile of loose wooden boards indoors.

Figures

From Figure 1: RMA on outdoor terrain: sand, mud, hiking trails, tall grass, a dirt pile, trail stairs, and cement an... — From Figure 1: RMA on outdoor terrain: sand, mud, hiking trails, tall grass, a dirt pile, trail stairs, and cement and pebble piles. Every run uses the same policy with no simulation calibration or real-world fine-tuning.

From Figure 1: The two RMA subsystems. In phase 1 the base policy is trained with RL in simulation, using an encoder ... — From Figure 1: The two RMA subsystems. In phase 1 the base policy is trained with RL in simulation, using an encoder that compresses privileged environment factors into a latent extrinsics vector. In phase 2 an adaptation module learns to predict the extrinsics from recent states and actions. At deployment the adaptation module runs at 10 Hz and the base policy at 100 Hz, always using the latest extrinsics estimate.

From Figure 2: Indoor tests comparing RMA, RMA without adaptation and A1's own controller on uneven foam, an incline,... — From Figure 2: Indoor tests comparing RMA, RMA without adaptation and A1's own controller on uneven foam, an incline, a mattress, step-ups and step-downs, with payload limits at the bottom. RMA steps down 15 cm with 80% success and crosses the mattress and uneven foam every time; A1's controller starts degrading at 8 kg of payload. Averages over 5 trials.

From Figure 3: The robot crossing an oily plastic sheet with plastic-covered feet. Knee torque, foot-contact pattern ... — From Figure 3: The robot crossing an oily plastic sheet with plastic-covered feet. Knee torque, foot-contact pattern and two components of the predicted extrinsics shift when it slips. After adapting, the gait period is close to the original, torque is higher and the extrinsics still signal a slippery surface.

Abstract

Successful real-world deployment of legged robots would require them to adapt in real-time to unseen scenarios like changing terrains, changing payloads, wear and tear. This paper presents Rapid Motor Adaptation (RMA) algorithm to solve this problem of real-time online adaptation in quadruped robots. RMA consists of two components: a base policy and an adaptation module. The combination of these components enables the robot to adapt to novel situations in fractions of a second. RMA is trained completely in simulation without using any domain knowledge like reference trajectories or predefined foot trajectory generators and is deployed on the A1 robot without any fine-tuning. We train RMA on a varied terrain generator using bioenergetics-inspired rewards and deploy it on a variety of difficult terrains including rocky, slippery, deformable surfaces in environments with grass, long vegetation, concrete, pebbles, stairs, sand, etc. RMA shows state-of-the-art performance across diverse real-world as well as simulation experiments. Video results at https://ashish-kmr.github.io/rma-legged-robots/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong