Learning Latent Representations to Influence Multi-Agent Interaction

Annie Xie, Dylan P. Losey, Ryan Tolsma, Chelsea Finn, Dorsa Sadigh

Seminal November 2020

TL;DR LILI learns a latent embedding of another agent's changing strategy from the ego agent's own observations, plus how that strategy reacts to the ego agent, and trains an RL policy that steers the other agent toward strategies it can exploit. In real robot air hockey it blocks 91% of shots against an opponent that keeps aiming away from it, versus 44% for SAC.

What they do

The other agent in a repeated interaction (a driver, an air hockey opponent) changes its policy in response to the robot. LILI (Learning and Influencing Latent Intent) skips modeling the other agent’s actions. It learns a latent strategy that is fixed within an interaction and changes between interactions, and how the robot’s behavior drives that change. Optimizing reward across interactions then makes the robot steer the other agent toward useful strategies.

How it works

Problem. Each interaction is a hidden-parameter MDP whose dynamics and reward depend on the unobserved strategy z. The next z depends on the current z and the ego agent’s trajectory.
Representation. An encoder maps interaction i’s (s, a, s’, r) tuples to the z of interaction i+1. A decoder reconstructs interaction i+1’s transitions and rewards from z, so no strategy labels are needed.
Policy. SAC actor and critic conditioned on state and z. The encoder also gets critic-loss gradients.
Influence. The objective sums discounted reward across interactions, so influence emerges without an extra reward. LILI (no influence) optimizes one interaction at a time.

Results

Point Mass: LILI overshoots or undershoots on purpose to keep the hidden target near its start, almost matching an Oracle that sees it. The best baseline, LILAC, moves to the circle’s center.
Lunar Lander, 2D driving and CARLA: LILI beats SAC, SLAC and LILAC and nearly matches the Oracle.
Air hockey with two Franka arms: over the last 100 interactions a random blocker stops 18% of shots, SAC 44%, LILI 91%. With a bonus for left blocks, LILI gets the striker to shoot left 41% of the time.
Against an expert human, LILI (no influence) blocks 29/40 shots, SAC 18/40.

Limitations

Full LILI was too brittle against humans: it blocked with the paddle’s edges and missed imperfect shots, and the planned user study did not happen. Strategies must be inferable from a short history; with an 11-interaction history LILI does no better than SAC.

Figures

From Figure 1: The ego robot learns air hockey against a robot that changes its aim between interactions, aiming away... — From Figure 1: The ego robot learns air hockey against a robot that changes its aim between interactions, aiming away from where the ego robot last blocked. The ego robot represents each opponent policy as a latent strategy z and uses how z changes to guide the opponent into aiming where it blocks best.

From Figure 2: Left, the ego agent uses its previous interaction to predict the other agent's current latent strategy... — From Figure 2: Left, the ego agent uses its previous interaction to predict the other agent's current latent strategy z and follows a policy conditioned on it. Right, from pairs of consecutive interactions, an encoder and decoder learn to predict the next interaction from the previous one, while model-free RL maximizes long-term reward.

From Figure 4: Point Mass. Left, reward per interaction during learning. Right, where the hidden target sat during th... — From Figure 4: Point Mass. Left, reward per interaction during learning. Right, where the hidden target sat during the final 500 interactions. Ending inside or outside the circle moves the target clockwise or counterclockwise; LILI uses this to keep the target near its start location.

From Figure 6: Air hockey. Left, block success rate during learning. Right, how often the opponent fired left, middle... — From Figure 6: Air hockey. Left, block success rate during learning. Right, how often the opponent fired left, middle or right in the final 200 interactions. Blocking left earns a bonus, and only LILI shifts the opponent toward left shots (41%).

Abstract

Seamlessly interacting with humans or robots is hard because these agents are non-stationary. They update their policy in response to the ego agent's behavior, and the ego agent must anticipate these changes to co-adapt. Inspired by humans, we recognize that robots do not need to explicitly model every low-level action another agent will make; instead, we can capture the latent strategy of other agents through high-level representations. We propose a reinforcement learning-based framework for learning latent representations of an agent's policy, where the ego agent identifies the relationship between its behavior and the other agent's future strategy. The ego agent then leverages these latent dynamics to influence the other agent, purposely guiding them towards policies suitable for co-adaptation. Across several simulated domains and a real-world air hockey game, our approach outperforms the alternatives and learns to influence the other agent.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong