Game-Theoretic Modeling of Human Adaptation in Human-Robot Collaboration

Stefanos Nikolaidis, Swaprava Nath, Ariel D. Procaccia, Siddhartha Srinivasa

Seminal January 2017

TL;DR The robot models its human teammate as learning the true payoffs of only the robot action just played, and plans when to fail on purpose to reveal its limits. In a 60-person table-clearing study this gave 42-52% more team reward than a robot that assumes the human adapts completely.

What they do

People often misjudge what a robot can do. This paper models how that belief changes as the person watches the robot act. The robot uses the model to choose each round between revealing one of its limits and taking the best action given what the human currently believes.

How it works

Repeated game. Robot and human share one reward matrix over action pairs. The robot knows the true matrix; the human best-responds to each robot action using their own, possibly wrong, one.
Partial adaptation. After the robot plays an action, the human learns that action’s row of true payoffs with probability alpha. The complete-adaptation baseline assumes the human learns every row.
Three models. The human learns before acting (M1), or after acting, with the robot observing it (M2) or inferring it from later actions (M3).
Planning. M3 is an MDP with 3^m states for m robot actions. Under M1 and M2, once the best revealed row is known the robot should keep playing it, which gives an O(mT) algorithm.

Results

60 participants, 3 rounds, alpha = 0.9. Partial adaptation earned 42% more reward than complete adaptation under partial observability and 52% more under full observability (F(1,56) = 18.58, p < 0.001). Observability had no significant effect.
The partial-adaptation robot fails at lifting both bottles in round 1, which teaches the human to empty the bottle. The complete-adaptation robot starts with the closest bottle; in round 2, 6 of its 10 participants did not empty the bottle.
In simulation over 1000 random reward matrices, partial adaptation wins across a wide range of horizons, with the gap shrinking for large T.

Limitations

The study uses one task. Alpha is hand-set, shared by all actions, and depends on task and population: only 73% of participants learned in round 2. Each action reveals only its own row, and the partially observable case scales exponentially.

Figures

From Figure 1: A user repeats a table-clearing task with the robot. Early in the task the robot fails on purpose to r... — From Figure 1: A user repeats a table-clearing task with the robot. Early in the task the robot fails on purpose to reveal its capabilities to the human teammate.

From Figure 2: Models of partial adaptation. When the robot plays a row, the human learns that row of the true reward... — From Figure 2: Models of partial adaptation. When the robot plays a row, the human learns that row of the true reward matrix R with probability alpha, and nothing with probability 1 - alpha. Learning happens before her action (M1) or after it (M2 with full observability, M3 with partial observability).

From Figure 4: Left, accumulated reward over 3 rounds for all four conditions. Center, predicted and actual reward pe... — From Figure 4: Left, accumulated reward over 3 rounds for all four conditions. Center, predicted and actual reward per round for partial and complete adaptation policies under partial observability. Right, mean reward over horizon T in simulation; the gain from partial adaptation shrinks for large T.

Abstract

In human-robot teams, humans often start with an inaccurate model of the robot capabilities. As they interact with the robot, they infer the robot's capabilities and partially adapt to the robot, i.e., they might change their actions based on the observed outcomes and the robot's actions, without replicating the robot's policy. We present a game-theoretic model of human partial adaptation to the robot, where the human responds to the robot's actions by maximizing a reward function that changes stochastically over time, capturing the evolution of their expectations of the robot's capabilities. The robot can then use this model to decide optimally between taking actions that reveal its capabilities to the human and taking the best action given the information that the human currently has. We prove that under certain observability assumptions, the optimal policy can be computed efficiently. We demonstrate through a human subject experiment that the proposed model significantly improves human-robot team performance, compared to policies that assume complete adaptation of the human to the robot.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong