On the Utility of Learning about Humans for Human-AI Coordination

Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, Anca Dragan

Seminal October 2019

TL;DR In a simplified two-player Overcooked kitchen, agents trained only with themselves (self-play, population-based training, joint planning) coordinate much worse with humans than a PPO agent trained against a behavior-cloned human model. In a study with 60 users, the human-aware agent scored significantly higher than self-play (p=.01) and PBT (p<.01).

What they do

Self-play works in competitive games because a human’s suboptimal move only helps the agent. In a common-payoff game the same mistake hurts the team, and co-trained agents can settle on conventions humans don’t share. The authors test this in a simplified two-chef Overcooked kitchen with five layouts designed to make motion or strategy coordination hard.

How it works

Task. Put 3 onions in a pot, wait 20 timesteps, plate and serve the soup for a shared reward of 20. Teams are scored over 400 timesteps.
Human models. About 16 human-human games per layout (18k timesteps in total) are split into two halves. Behavior cloning on a hand-designed 64-D state featurization gives a training model (BC) and a held-out test partner (H_Proxy).
Agents trained with themselves. PPO self-play (SP), population-based training (PBT) with 3 agents, and a joint planner that replans after every partner action.
Agents trained for humans. PPO with BC embedded in the environment (PPO_BC), annealed from self-play, and an A* planner that queries the partner model. As a gold standard, both are also trained with H_Proxy itself.

Results

With H_Proxy, SP and PBT do very well with themselves but struggle to match the pure imitation agent BC. PPO_BC outperforms them and gets closer to the gold standard.
On the 2 layouts where planning was feasible, the joint planner drops far below the gold standard with H_Proxy; much of the drop seems to come from expecting optimal motion. Planning with BC instead of the exact model gets stuck in loops.
60 Mechanical Turk users, between-subjects: PPO_BC beats SP and PBT in three layouts and roughly matches the better one in the other two. Agent type has a significant effect (F(2,224)=6.49, p<.01). With non-cooperative players kept in, PPO_BC vs SP falls to p=.06.
SP and PBT predict real human actions poorly, while BC and H_Proxy do decently.

Limitations

The human model is plain, non-adaptive behavior cloning; it is very poor on Forced Coordination and Counter Circuit, which the authors suspect is why PPO_BC helps less there. They guess real users adapt to the agent, which the stationary model cannot capture. Each agent is trained for one layout, and the authors cannot fully rule out that BC helps only by exposing the agent to more varied states.

Figures

From Figure 1: Wrongly expecting the human to be optimal. In a competitive game (left), the AI plans for the worst ca... — From Figure 1: Wrongly expecting the human to be optimal. In a competitive game (left), the AI plans for the worst case, so a suboptimal human move only raises its reward (7 instead of the expected 3). In a collaborative game (right), the AI expects the human to coordinate on the best option (8), and a suboptimal human move gives it 1, the worst outcome.

From Figure 2: The Overcooked environment. Players put three onions in a pot, plate the cooked soup and deliver it, a... — From Figure 2: The Overcooked environment. Players put three onions in a pot, plate the cooked soup and deliver it, as many times as possible. Here the optimal split is for the human to add an onion and the AI to deliver the soup, but the human is heading for a plate; an AI that expects the human to play optimally keeps going for its own dish and coordination fails.

From Figure 4: Reward per episode with the proxy human model, comparing against agents trained in self-play. Self-pla... — From Figure 4: Reward per episode with the proxy human model, comparing against agents trained in self-play. Self-play agents (SP) score high with themselves (dotted bars) but drop when paired with the proxy human. PPO trained with the BC human model (orange) does better and comes closer to the gold standard, PPO trained with the proxy itself (red dotted line).

From Figure 6: Average reward over 400 timesteps with real users, with standard error across participants. In most la... — From Figure 6: Average reward over 400 timesteps with real users, with standard error across participants. In most layouts the PPO agent trained with human data (orange) beats the agents that do not model the human (SP and PBT, teal); human-human teams are shown in gray.

Abstract

While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Agents that assume their partner to be optimal or similar to them can converge to coordination protocols that fail to understand and be understood by humans. To demonstrate this, we introduce a simple environment that requires challenging coordination, based on the popular game Overcooked, and learn a simple model that mimics human play. We evaluate the performance of agents trained via self-play and population-based training. These agents perform very well when paired with themselves, but when paired with our human model, they are significantly worse than agents designed to play with the human model. An experiment with a planning algorithm yields the same conclusion, though only when the human-aware planner is given the exact human model that it is playing with. A user study with real humans shows this pattern as well, though less strongly. Qualitatively, we find that the gains come from having the agent adapt to the human's gameplay. Given this result, we suggest several approaches for designing agents that learn about humans in order to better coordinate with them. Code is available at https://github.com/HumanCompatibleAI/overcooked_ai.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong