Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel

★ Must read March 2017

TL;DR Train an object localizer only on simulated images with random non-realistic textures, lighting, camera poses and distractors. In the real world it localizes objects to about 1.5 cm and lets a Fetch robot pick targets out of clutter in 38 of 40 trials.

What they do

Instead of making a simulator look realistic, they randomize its rendering so heavily that the real world looks like one more variation. They test this on object localization: a CNN, trained only on MuJoCo renders, maps one frame from an uncalibrated webcam to the (x, y, z) position of a target object on a table.

How it works

Randomization. Every training image draws 0-10 distractors, object positions, textures for everything (random color, gradient or checker), lights, image noise, and the camera position (within a 10x5x10 cm box), orientation and field of view.
No color cue. Object textures are random, so the detector knows only the target’s size and shape.
Model. VGG-16 convolutional layers with smaller fully connected layers (256 and 64), trained with Adam on an L2 position loss. Table height is fixed, so the task is effectively 2D. One detector is trained per object.

Results

On 480 real webcam images of eight geometric objects, average error is about 1.5 cm, with per-object errors from 0.6 to 3.2 cm across object-only, distractor and occlusion conditions. Error in simulation is 0.3-0.5 cm, so the detectors still overfit.
Ablations: error rises sharply below 1,000 unique textures. Training without distractors raises error on cluttered scenes from 1.8 to 7.2 cm; dropping camera randomization costs 0.5-0.7 cm; image noise has negligible effect. With enough data, training from scratch nearly matches ImageNet pretraining, which mainly helps when data is scarce.
On a Fetch robot with off-the-shelf motion planning, two detectors picked the target in 38 of 40 increasingly cluttered scenes. A detector for a YCB Spam can succeeded in 9 of 10 trials among unseen food-item distractors.

Limitations

Only position is estimated, not orientation, and it relies on a fixed table height and one monocular camera. The grasp itself is prescribed and planned, not learned. The authors leave contact-rich and higher-precision tasks to future work.

Figures

From Figure 1: A detector is trained on low-fidelity renders with random camera positions, lighting, object positions... — From Figure 1: A detector is trained on low-fidelity renders with random camera positions, lighting, object positions and non-realistic textures, then used on real images with no additional training.

From Figure 2: The network takes a webcam image downsized to 224x224, passes it through VGG-style convolutional layer... — From Figure 2: The network takes a webcam image downsized to 224x224, passes it through VGG-style convolutional layers and two small fully connected layers, and outputs the (x, y, z) position of the object.

From Figure 5: Real-world localization error against the number of unique textures and lighting conditions, each dete... — From Figure 5: Real-world localization error against the number of unique textures and lighting conditions, each detector trained on 10,000 images.

From Figure 6: Two grasping runs: the detector, trained only in simulation, estimates the target position and a motio... — From Figure 6: Two grasping runs: the detector, trained only in simulation, estimates the target position and a motion planner grasps the object there.

Abstract

Bridging the 'reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to $1.5$cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong