Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience

Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, Dieter Fox

Seminal October 2018

TL;DR SimOpt alternates RL training in a GPU simulator with updates to the distribution of simulation parameters, so that simulated roll-outs of the policy match a few real-robot roll-outs. After two updates with 3 real roll-outs each, a Yumi swings a peg on a rope into a hole in 90% of 20 trials; after one update, a Panda opens a drawer in all 20 trials.

What they do

Domain randomization needs hand-tuned parameter ranges, and ranges that are too wide include infeasible scenes or yield overly conservative policies. SimOpt starts from a conservative distribution over simulation parameters and adapts it with a few real-world roll-outs, until the trained policy behaves the same in simulation and on the real robot. It does not try to replicate the real scene exactly, and rewards are computed only in simulation.

How it works

  • Loop. Train a policy with PPO on the current parameter distribution in NVIDIA Flex, run it a few times on the real robot, update the distribution, and repeat. Policy weights carry over between iterations.
  • Discrepancy. Weighted L1 and L2 distances between simulated and real observation trajectories (joint angles plus an object position tracked with DART), with Gaussian smoothing for misalignment. Only partial real-world observations are needed.
  • Update. The parameters follow a full-covariance Gaussian. The simulator is not differentiable, so a sampling-based method built on relative entropy policy search updates it under a KL-divergence bound, which keeps the new distribution close to what the current policy was trained on.
  • Scale. 64 GPUs with 150 simulated agents each.

Results

  • Simulated drawer opening: randomizing the cabinet position with a 2 cm standard deviation gives a policy that opens the drawer; with 10 cm it reaches the handle but does not open it. SimOpt transferred to a cabinet offset by 15 cm in 3 RL+SimOpt iterations, and by 22 cm in 5.
  • Swing-peg-in-hole (ABB Yumi, peg on a soft rope): each iteration is about 7 min of RL, 3 real roll-outs and 3 updates of 9600 samples. After two iterations: 90% success over 20 trials. The largest changes were in rope parameters and action scaling.
  • Drawer opening (Franka Panda): before the update, one finger pushes the handle until the gripper opens; after one update, the drawer opens in all 20 trials.

Limitations

The parameter distribution is a single Gaussian; multi-modal distributions are left to future work. Real-world tracking with DART needs depth images and 3D articulated object models. Policies observe only joint angles and object positions, with no vision or touch. On the real robots, results are only compared before and after SimOpt updates; the comparison to standard domain randomization is done in simulation.

Figures

From Figure 2: The SimOpt loop. A policy trained on the current simulation parameter distribution is run in the real ...
From Figure 2: The SimOpt loop. A policy trained on the current simulation parameter distribution is run in the real world and across sampled parameters in simulation; the discrepancy between simulated and real observations updates the distribution.
From Figure 7: Real-robot runs at successive SimOpt iterations. Left: swing-peg-in-hole succeeds after two iterations...
From Figure 7: Real-robot runs at successive SimOpt iterations. Left: swing-peg-in-hole succeeds after two iterations that adjust the rope, peg and robot parameters. Right: before the update, one finger pushes too hard on the drawer handle and the gripper opens; after one iteration the robot keeps its gripper oriented and opens the drawer.
From Figure 5: In simulation, the distribution of the cabinet position starts at the source scene (far left) and move...
From Figure 5: In simulation, the distribution of the cabinet position starts at the source scene (far left) and moves toward the target scene over 5 SimOpt iterations.
Abstract

We consider the problem of transferring policies to the real world by training on a distribution of simulated scenarios. Rather than manually tuning the randomization of simulations, we adapt the simulation parameter distribution using a few real world roll-outs interleaved with policy training. In doing so, we are able to change the distribution of simulations to improve the policy transfer by matching the policy behavior in simulation and the real world. We show that policies trained with our method are able to reliably transfer to different robots in two real world tasks: swing-peg-in-hole and opening a cabinet drawer. The video of our experiments can be found at https://sites.google.com/view/simopt

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.