Deep reinforcement learning from human preferences

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei

★ Must read June 2017

TL;DR A human picks the better of two short clips of the agent, a neural reward model is fit to those choices, and RL optimizes that learned reward. With 700 comparisons it nearly matches RL on the true reward across eight MuJoCo tasks, and it learns a Hopper backflip from 900 comparisons in under an hour.

What they do

Many goals are hard to write as a reward or to demonstrate. Here a non-expert human instead watches pairs of short clips of the agent and picks the better one. A reward model is fit to these comparisons and deep RL optimizes it, with human feedback on less than 1% of the agent’s interactions. Tests use MuJoCo robotics tasks and Atari games with the true reward hidden.

How it works

Three asynchronous loops. TRPO (MuJoCo) or A2C (Atari) maximizes the predicted reward; a human labels clip pairs from its rollouts as one better, a tie, or “can’t tell”; the reward model is refit to all labels so far.
Preference model. Bradley-Terry: the chance a clip is preferred grows exponentially with its summed predicted reward, plus a 10% chance of a random answer.
Queries. An ensemble of three bootstrapped predictors; the human sees the candidate pairs where they disagree most.
No leaked supervision. Variable-length episodes are removed and Atari score displays blanked out.

Results

MuJoCo: 700 labels nearly match RL on the true reward on all eight tasks; 1400 synthetic labels do slightly better. Human feedback was half to equally as efficient as synthetic, and better on Ant.
Atari (5,500 human labels): mostly similar to or slightly worse than the same number of synthetic labels; beats A3C on Enduro, fails to clear Qbert’s first level.
Novel behaviors, each from under an hour of feedback: Hopper backflips (900 queries) and a one-legged Half-Cheetah (800).
Ablation: a reward model trained only on early queries does poorly; on Pong the agent avoids losing points without scoring.

Limitations

Simulation only, with environments modified to remove termination signals. Learned-reward training is less stable than RL on the true reward and trails it on several Atari games. Disagreement-based query selection hurts on some tasks. The novel behaviors were trained with the authors’ own feedback.

Figures

From Figure 1: The reward predictor is trained asynchronously from human comparisons of trajectory segments, and the ... — From Figure 1: The reward predictor is trained asynchronously from human comparisons of trajectory segments, and the RL agent maximizes the predicted reward.

From Figure 2: True reward on the MuJoCo tasks for real human feedback (purple), synthetic oracle feedback (blue) and... — From Figure 2: True reward on the MuJoCo tasks for real human feedback (purple), synthetic oracle feedback (blue) and RL on the true reward (orange). Synthetic curves average 5 runs; the human-feedback curve is a single run.

From Figure 4: Four frames from a single backflip; the Hopper is trained to perform a sequence of backflips, landing ... — From Figure 4: Four frames from a single backflip; the Hopper is trained to perform a sequence of backflips, landing upright each time.

Abstract

For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong