Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

HG-DAgger: Interactive Imitation Learning with Human Experts

Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, Mykel J. Kochenderfer

Seminal October 2018

TL;DR A DAgger variant where the human expert decides when to take over from the learner and keeps full control until handing back, with labels recorded only during those takeovers. On a real car weaving between obstacles, the HG-DAgger policy had no collisions or road departures, against 1 collision and 6 departures for behavioral cloning.

What they do

DAgger collects expert labels while control switches at random between the expert and the learner (the novice). With a human expert this hurts safety and label quality: the human gets little feedback from their own inputs and perceives actuator lag. In HG-DAgger (Human-Gated DAgger) the human decides when to take over and keeps uninterrupted control until handing back. It is tested on weaving a car between stationary obstacles, in simulation and on a real vehicle.

How it works

  • Human gating. The novice drives until the expert judges the state unsafe, takes over (on the car, by turning the wheel), recovers, then returns control with a button. Only these recovery segments are labeled and added to a dataset that starts from behavioral cloning data.
  • Doubt. The novice is a neural-network ensemble; its doubt is the l2 norm of the diagonal of the ensemble’s output covariance.
  • Learned threshold. Doubt is logged at each intervention, and the threshold is the mean of the last 25% of those values. States below it form the estimated safe set.
  • Setup. A low-dimensional state (lane position, heading, speed, distances to lane edges and obstacles) maps to steering and speed. Every method starts from 10,000 cloning labels and adds 2,000 per epoch for five epochs.

Results

  • Simulation: HG-DAgger learns faster and more stably than DAgger and behavioral cloning; DAgger turns unstable in later epochs as its expert share decays.
  • Rollouts started inside the estimated safe set had a 12x lower collision rate and 20x lower road-departure rate than those started outside; departures lasted 1.63 s vs 3.74 s.
  • Real car: 0 collisions and 0 road departures, versus 1 and 6 for behavioral cloning and 1 and 1 for DAgger. Steering was 21.1% closer to the human’s distribution than DAgger’s.
  • Over 40 obstacle layouts, the learned threshold was near-optimal for separating free from occupied space.

Limitations

The authors say the small amount of on-vehicle test data limits statistical significance. Obstacles on the real car were simulated, the policy sees a hand-designed state, and the risk threshold is only used for evaluation, not to gate the policy at test time. Training-time safety relies on the human intervening.

Figures

From Figure 2: Test vehicle (left) and expert driver interface (right).
From Figure 2: Test vehicle (left) and expert driver interface (right).
From Figure 5: On-vehicle test trajectories of the behavioral cloning policy.
From Figure 5: On-vehicle test trajectories of the behavioral cloning policy.
From Figure 5: On-vehicle test trajectories of the HG-DAgger policy.
From Figure 5: On-vehicle test trajectories of the HG-DAgger policy.
From Figure 6: Risk maps for a policy trained on the test vehicle. The center map uses the threshold learned from hum...
From Figure 6: Risk maps for a policy trained on the test vehicle. The center map uses the threshold learned from human interventions. The purple box is the ego vehicle and white boxes are other vehicles; blue is safe and red is unsafe.
Abstract

Imitation learning has proven to be useful for many real-world problems, but approaches such as behavioral cloning suffer from data mismatch and compounding error issues. One attempt to address these limitations is the DAgger algorithm, which uses the state distribution induced by the novice to sample corrective actions from the expert. Such sampling schemes, however, require the expert to provide action labels without being fully in control of the system. This can decrease safety and, when using humans as experts, is likely to degrade the quality of the collected labels due to perceived actuator lag. In this work, we propose HG-DAgger, a variant of DAgger that is more suitable for interactive imitation learning from human experts in real-world systems. In addition to training a novice policy, HG-DAgger also learns a safety threshold for a model-uncertainty-based risk metric that can be used to predict the performance of the fully trained novice in different regions of the state space. We evaluate our method on both a simulated and real-world autonomous driving task, and demonstrate improved performance over both DAgger and behavioral cloning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.