Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Generative Adversarial Imitation Learning

Jonathan Ho, Stefano Ermon

Seminal June 2016

TL;DR Learn a policy straight from expert demonstrations by training it against a discriminator that tells its state-action pairs from the expert's, with no reward and no IRL inner loop. On the MuJoCo tasks it almost always reaches at least 70% of expert performance, and on Humanoid it matches the expert, where behavioral cloning stays under 60%.

What they do

The setting is imitation from expert trajectories alone, with no expert queries and no reward. Behavioral cloning suffers from compounding errors, and inverse RL runs RL in an inner loop only to return a cost. The paper shows that RL on a cost from regularized max-causal-entropy IRL yields the policy whose occupancy measure (its distribution of visited state-action pairs) is closest to the expert’s. The regularizer sets the distance. One choice makes it the Jensen-Shannon divergence and leads to a GAN-style algorithm that learns the policy directly.

How it works

  • IRL as occupancy matching. Without regularization, RL after IRL exactly reproduces the expert’s occupancy measure. Earlier apprenticeship-learning methods limit costs to hand-designed linear features, so they imitate well only when those features fit.
  • GAN regularizer. The new regularizer turns the objective into the best log loss of a classifier that separates policy pairs from expert pairs. Up to a constant, that loss is the Jensen-Shannon divergence. An optional causal-entropy term, weighted by λ, is added.
  • Algorithm. A discriminator D(s,a) is trained with Adam, and the policy takes TRPO steps with log D(s,a) as its cost.

Results

  • There are 9 tasks: cartpole, acrobot, mountain car and six MuJoCo tasks. The experts are TRPO policies, and the baselines are behavioral cloning, FEM and GTAL.
  • On the classic control tasks, GAIL beat every baseline.
  • On the MuJoCo tasks except Reacher, it almost always reached at least 70% of expert performance. On Humanoid it matched the expert while behavioral cloning stayed under 60%. On Ant, FEM and GTAL did worse than a random policy.
  • On Reacher, behavioral cloning was more sample-efficient than GAIL.

Limitations

GAIL needs little expert data but a lot of environment interaction: about as many samples as TRPO needed to train the experts from reward. Because it is model-free, it generally needs more interaction than model-based methods. It also explores randomly, where DAgger-style methods can ask the expert which action to take.

Figures

From Algorithm 1: each iteration samples policy trajectories, takes a gradient step on the discriminator to separate ...
From Algorithm 1: each iteration samples policy trajectories, takes a gradient step on the discriminator to separate them from expert pairs, then takes a TRPO policy step with cost log D(s,a).
From Figure 1: (a) Performance of learned policies against the number of expert trajectories, with negative cost scal...
From Figure 1: (a) Performance of learned policies against the number of expert trajectories, with negative cost scaled so the expert scores 1 and a random policy 0. (b) The effect of the causal entropy weight λ on Reacher.
Abstract

Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.