Sim-to-Real Transfer of Robotic Control with Dynamics Randomization

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel

Seminal October 2017

TL;DR Train a recurrent puck-pushing policy in MuJoCo with 95 dynamics parameters randomized each episode, then run it on a real Fetch arm with no real-world training. The LSTM policy succeeds 0.89 of the time on the real robot versus 0.91 in simulation.

What they do

Policies trained in simulation tend to exploit the simulator’s quirks and fail on the real robot. Instead of calibrating the simulator carefully, the authors randomize its dynamics during training and give the policy memory, so it can infer the dynamics it is facing at run time. They test this on pushing a puck to a random target with a 7-DoF Fetch arm, trained only in MuJoCo with a sparse binary reward.

How it works

Dynamics randomization. 95 parameters are randomized in total. Link masses, joint damping, puck mass, friction and damping, table height and controller gains are sampled once per episode. The time between actions (a simple latency model) and Gaussian observation noise change every step.
Recurrent policy. An LSTM branch reads the current state and previous action to implicitly identify the dynamics. A feedforward branch handles the goal. Actions are relative joint-angle targets for a position controller.
Training. Recurrent DDPG with Hindsight Experience Replay. The critic also sees the true dynamics parameters, which the policy never gets. Training takes about 100 million samples, roughly 8 hours on a 100-core cluster.

Results

Real-robot success: LSTM 0.89 ± 0.06 (28 trials), feedforward with an 8-step history 0.70, plain feedforward 0.67, feedforward without randomization 0.0. In simulation the LSTM scores 0.91.
Ablations on the real robot: a fixed action timestep drops success to 0.29, no observation noise to 0.25, fixed puck friction to 0.48 and fixed link mass to 0.64.
With a bag of chips taped under the puck to change its contact dynamics, the LSTM still succeeds 0.91 ± 0.04.

Limitations

Only one task, puck pushing, was tested. The policy works from full state, with the puck tracked by a PhaseSpace mocap system rather than vision. Real-robot evaluations use 10 to 28 trials per model. The authors leave other tasks and vision inputs to future work.

Videos

Figures

From Figure 1: A recurrent policy trained for pushing in simulation, deployed directly on a Fetch arm. The red marker... — From Figure 1: A recurrent policy trained for pushing in simulation, deployed directly on a Fetch arm. The red marker is the puck's target location.

From Figure 4: The policy network. Inputs needed to infer the dynamics (state and previous action) go through the rec... — From Figure 4: The policy network. Inputs needed to infer the dynamics (state and previous action) go through the recurrent LSTM branch; the goal and a copy of the state go through the feedforward branch.

From Figure 7: Pushing success in simulation and on the real robot for each architecture, all trained only in simulat... — From Figure 7: Pushing success in simulation and on the real robot for each architecture, all trained only in simulation.

Abstract

Simulations are attractive environments for training agents as they provide an abundant source of data and alleviate certain safety concerns during the training process. But the behaviours developed by agents in simulation are often specific to the characteristics of the simulator. Due to modeling error, strategies that are successful in simulation may not transfer to their real world counterparts. In this paper, we demonstrate a simple method to bridge this "reality gap". By randomizing the dynamics of the simulator during training, we are able to develop policies that are capable of adapting to very different dynamics, including ones that differ significantly from the dynamics on which the policies were trained. This adaptivity enables the policies to generalize to the dynamics of the real world without any training on the physical system. Our approach is demonstrated on an object pushing task using a robotic arm. Despite being trained exclusively in simulation, our policies are able to maintain a similar level of performance when deployed on a real robot, reliably moving an object to a desired location from random initial configurations. We explore the impact of various design decisions and show that the resulting policies are robust to significant calibration error.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong