Enabling Robots to Communicate their Objectives

Sandy H. Huang, David Held, Pieter Abbeel, Anca D. Dragan

Seminal February 2017

TL;DR A simulated self-driving car chooses which of its own behaviors to show people so they learn its objective, modeling the viewer as an approximate inverse-RL learner rather than an exact one. With the best learner model plus examples covering every driving strategy, Mechanical Turk users predicted the car's behavior in new scenes significantly better than after random examples (p = 0.049).

What they do

People anticipate a robot better if they understand the objective it optimizes. The robot here picks which environments to show its optimal behavior in, so that a person watching infers the right objective. This is inverse reinforcement learning with the roles swapped: the robot models the human as an IRL learner. The twist is that humans are assumed to infer approximately, not exactly. The domain is a simulated autonomous car on a three-lane highway.

How it works

Learner models. The reward is a weighted sum of five driving features. Exact-inference IRL eliminates every candidate weighting under which the shown trajectory is not optimal. The approximate models compare the shown trajectory with each candidate’s optimal one by reward, Euclidean or strategy distance. They then either eliminate far candidates (deterministic) or down-weight them (probabilistic).
Example selection. Greedily pick up to ten of 21,216 environments to maximize the modeled probability of the true weights.
Coverage. Once extra examples stop helping, add the most informative example from each driving strategy not yet shown (e.g. merging in front as well as merging behind).

Results

161 Mechanical Turk participants (after filtering) picked the car’s trajectory among four in six test environments. The best model, deterministic Euclidean (“approx*”), beat exact-inference IRL (p = 0.025). Exact IRL showed a single example.
Without coverage, approx* did not significantly beat a median random sequence of eight examples (p = 0.23).
Example count did not correlate with performance (r = 0.03); the number of helpful examples per environment class did (r = 0.83).
Coverage-approx* significantly beat random (p = 0.049); coverage-random did not (p = 0.159).

Limitations

Only simulated driving with one other car and a fixed true objective is tested. Most approximate models did no better than exact IRL, and the effect of coverage suggests an IRL-only model does not capture how people extrapolate from robot behavior.

Figures

From Figure 1: When the yellow car must merge right, its behavior says more about whether it is aggressive or defensi... — From Figure 1: When the yellow car must merge right, its behavior says more about whether it is aggressive or defensive if another car is present (left) than if the lane is empty (right).

From Figure 2: The driving environments fall into four classes with two trajectory strategies each; the autonomous ca... — From Figure 2: The driving environments fall into four classes with two trajectory strategies each; the autonomous car is yellow, the other car gray, and the goal (merge right or drive forward) is highlighted in blue.

From Figure 5: Participants' confidence scores at identifying the car's trajectory in test environments, for each lea... — From Figure 5: Participants' confidence scores at identifying the car's trajectory in test environments, for each learner model (left) and with coverage added (right); coverage-approx* beats random significantly, coverage-random does not.

From Figure 6: Performance does not track the total number of examples shown (left) but does track the number of help... — From Figure 6: Performance does not track the total number of examples shown (left) but does track the number of helpful examples shown for an environment class (right).

Abstract

The overarching goal of this work is to efficiently enable end-users to correctly anticipate a robot's behavior in novel situations. Since a robot's behavior is often a direct result of its underlying objective function, our insight is that end-users need to have an accurate mental model of this objective function in order to understand and predict what the robot will do. While people naturally develop such a mental model over time through observing the robot act, this familiarization process may be lengthy. Our approach reduces this time by having the robot model how people infer objectives from observed behavior, and then it selects those behaviors that are maximally informative. The problem of computing a posterior over objectives from observed behavior is known as Inverse Reinforcement Learning (IRL), and has been applied to robots learning human objectives. We consider the problem where the roles of human and robot are swapped. Our main contribution is to recognize that unlike robots, humans will not be exact in their IRL inference. We thus introduce two factors to define candidate approximate-inference models for human learning in this setting, and analyze them in a user study in the autonomous driving domain. We show that certain approximate-inference models lead to the robot generating example behaviors that better enable users to anticipate what it will do in novel situations. Our results also suggest, however, that additional research is needed in modeling how humans extrapolate from examples of robot behavior.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong