Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences

Erdem Bıyık, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, Dorsa Sadigh

Seminal June 2020

TL;DR DemPref initializes a belief over a robot's reward from a few human demonstrations, then refines it with pairwise preference queries chosen by information gain. With one demonstration it reaches in 10 queries the reward alignment that preferences alone need 20-35 queries for, and users rated its queries significantly easier to answer than volume-removal queries.

What they do

Demonstrations are informative but noisy and hard to give on high-DoF robots; pairwise preferences are precise but carry at most one bit each. DemPref uses both: a few demonstrations set a prior over the reward, and actively chosen preference queries refine it. It also replaces the standard volume-removal query objective, which produces hard and sometimes useless questions, with information gain.

How it works

Reward and prior. The reward is linear in given trajectory features, r = ω·Φ(ξ). Demonstrations, modeled as Boltzmann-rational, give the initial belief over ω, represented by samples.
Queries. The robot shows two trajectories and the human picks one (strict) or may also answer “About Equal” (weak, with a minimum perceivable difference δ). Each answer updates the belief by Bayes’ rule.
Information gain. Volume removal is maximized by a trivial query with identical options. Information gain instead rewards queries where the robot cannot predict the answer but the human can answer confidently, at the same computational cost.
Theory. Demonstrations before queries yield at least as much information as the reverse order. Subtracting a query cost gives a stopping rule: stop when the best query’s value is negative.

Results

With one demonstration, 10 queries reach the alignment that preferences alone need 30 (Driver), 35 (Lunar Lander) and 20 (Fetch) queries for.
Physical Fetch study (15 users): DemPref with 1 demonstration and 15 queries beat Bayesian IRL with 5 demonstrations on task success and preferred system (p<0.05), but was not rated easier (p=0.297).
Information gain learned faster than volume removal in all four simulated domains. Users rated its queries easier in Driver, Tosser and Fetch (p<0.005) and in Driver and Tosser preferred the trajectories it learned (p<0.05).

Limitations

The reward is linear in hand-designed features, and only two feedback types are studied, against few baselines. Three lower-quality demonstrations did worse than one on Fetch. Users complained about the roughly 45 s it took to generate each query. Greedy information-gain selection has no optimality guarantee.

Videos

Figures

From Figure 1: A human expert can teach the robot through demonstrations or preferences; the robot selects which sour... — From Figure 1: A human expert can teach the robot through demonstrations or preferences; the robot selects which source to use and combines both to learn the reward.

From Figure 3: DemPref overview. Demonstrations initialize a distribution over the reward; the robot then generates t... — From Figure 3: DemPref overview. Demonstrations initialize a distribution over the reward; the robot then generates two trajectories, asks the human which they prefer, and updates the distribution.

From Figure 7: Alignment with the true reward versus number of preference queries on Driver, Lunar Lander and Fetch, ... — From Figure 7: Alignment with the true reward versus number of preference queries on Driver, Lunar Lander and Fetch, starting from 0, 1 or 3 demonstrations. One demonstration speeds up convergence significantly.

From Figure 11: User studies. (a) Information gain queries were rated easier than volume removal queries; (b) users f... — From Figure 11: User studies. (a) Information gain queries were rated easier than volume removal queries; (b) users found fewer indistinguishable options with information gain; (c) users preferred the trajectory learned with information gain.

Abstract

Reward functions are a common way to specify the objective of a robot. As designing reward functions can be extremely challenging, a more promising approach is to directly learn reward functions from human teachers. Importantly, data from human teachers can be collected either passively or actively in a variety of forms: passive data sources include demonstrations, (e.g., kinesthetic guidance), whereas preferences (e.g., comparative rankings) are actively elicited. Prior research has independently applied reward learning to these different data sources. However, there exist many domains where multiple sources are complementary and expressive. Motivated by this general problem, we present a framework to integrate multiple sources of information, which are either passively or actively collected from human users. In particular, we present an algorithm that first utilizes user demonstrations to initialize a belief about the reward function, and then actively probes the user with preference queries to zero-in on their true reward. This algorithm not only enables us combine multiple data sources, but it also informs the robot when it should leverage each type of information. Further, our approach accounts for the human's ability to provide data: yielding user-friendly preference queries which are also theoretically optimal. Our extensive simulated experiments and user studies on a Fetch mobile manipulator demonstrate the superiority and the usability of our integrated framework.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong