CAD2RL: Real Single-Image Flight without a Single Real Image

Fereshteh Sadeghi, Sergey Levine

Seminal November 2016

TL;DR Train a quadrotor's collision-avoidance policy from single RGB images entirely in CAD hallways with randomized textures and lighting, then fly it in real buildings. In a realistic scanned hallway it flies 1.2 km without a collision in about 40% of runs, and in real flights it collides 6 times over 164 m versus 13 for a free-space-prediction baseline.

What they do

Learning collision avoidance by trial and error is impractical on a fragile quadrotor. CAD2RL trains a vision-based policy entirely in simulated 3D hallways and flies it on real drones without ever seeing a real image. Instead of making the renderer realistic, it randomizes textures, lighting and furniture so heavily that real hallways look like one more variation.

How it works

Policy. A VGG16-based fully convolutional network maps one monocular RGB image to a 41x41 grid of bins; each bin is a flight direction through that part of the image, and the drone greedily picks the bin with the highest Q-value.
Training. The network is first pretrained to predict free space (a 1 m raycast per bin). It is then finetuned with Monte Carlo policy evaluation: from random states, every action is rolled out with the current policy for 5 steps, and the network regresses onto the resulting returns. Reward is 0 at a collision and otherwise grows with distance to the nearest obstacle, capped at 1.
Simulator. 24 Blender hallways from 12 floorplans, 21 furniture items and 200 wall textures, with random lights and camera height and yaw.

Results

In a realistic scanned hallway (60 fixed starts), CAD2RL keeps 1.2 km of collision-free flight in about 40% of runs, ahead of free-space prediction alone, a left-right-straight classifier and a straight-line controller.
Real hallway following on Parrot Bebop drones: 6 collisions over 164 m (27.3 m mean safe flight) versus 13 over 162 m (12.5 m) for free-space prediction in Cory Hall; 4 versus 7 collisions in SDH.
On 620 real RGB-D images, free-space prediction trained on randomized hallways beats fixed-texture training and training on a realistic reconstruction of the same building.

Limitations

The policy only avoids collisions, with no navigation goal, and still crashes in the real world: a reflective glass door fooled it, and particularly novel situations confused it. The authors expect that finetuning on real data, or adding sensors such as depth cameras, would improve it.

Videos

Figures

From Figure 1: Left, training only in simulation: rollouts of multiple actions from each state train a network to pre... — From Figure 1: Left, training only in simulation: rollouts of multiple actions from each state train a network to predict the long-horizon collision probability of each action. Right, the policy flying real hallways.

From Figure 2: Simulator renderings with randomized textures, lighting and furniture placement.

From Figure 5: Fraction of flights still collision-free versus distance flown in a realistically textured hallway; CA... — From Figure 5: Fraction of flights still collision-free versus distance flown in a realistically textured hallway; CAD2RL outperforms the prior L-R-S method and the other baselines.

From Figure 8: Free-space prediction on real images with networks trained in different simulators; randomized hallway... — From Figure 8: Free-space prediction on real images with networks trained in different simulators; randomized hallways beat less randomized simulations and even realistically textured hallways.

Abstract

Deep reinforcement learning has emerged as a promising and powerful technique for automatically acquiring control policies that can process raw sensory inputs, such as images, and perform complex behaviors. However, extending deep RL to real-world robotic tasks has proven challenging, particularly in safety-critical domains such as autonomous flight, where a trial-and-error learning process is often impractical. In this paper, we explore the following question: can we train vision-based navigation policies entirely in simulation, and then transfer them into the real world to achieve real-world flight without a single real training image? We propose a learning method that we call CAD$^2$RL, which can be used to perform collision-free indoor flight in the real world while being trained entirely on 3D CAD models. Our method uses single RGB images from a monocular camera, without needing to explicitly reconstruct the 3D geometry of the environment or perform explicit motion planning. Our learned collision avoidance policy is represented by a deep convolutional neural network that directly processes raw monocular images and outputs velocity commands. This policy is trained entirely on simulated images, with a Monte Carlo policy evaluation algorithm that directly optimizes the network's ability to produce collision-free flight. By highly randomizing the rendering settings for our simulated training set, we show that we can train a policy that generalizes to the real world, without requiring the simulator to be particularly realistic or high-fidelity. We evaluate our method by flying a real quadrotor through indoor environments, and further evaluate the design choices in our simulator through a series of ablation studies on depth prediction. For supplementary video see: https://youtu.be/nXBWmzFrj5s

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong