AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

Ye Yuan, Xinshuo Weng, Yanglan Ou, Kris Kitani

Seminal March 2021

TL;DR A Transformer that flattens all agents' trajectories into one sequence and uses agent-aware attention, so any agent's past state can directly inform another agent's future. It gets the best average FDE on ETH/UCY (0.39 m) and beats Trajectron++ on nuScenes (ADE10 1.45 m vs 1.51 m).

What they do

Multi-agent trajectory forecasters usually model time and social interaction in separate stages, for example an LSTM per agent followed by a graph network across agents. AgentFormer instead flattens the past and future states of all agents into one sequence and runs a single Transformer over it, so an agent’s state at one time can directly affect another agent’s state at a later time. It is used inside a stochastic CVAE predictor for pedestrians and vehicles.

How it works

Time encoder. A sinusoidal timestamp of each element’s timestep replaces positional encoding.
Agent-aware attention. A mask selects one set of query/key projections for elements of the same agent and another for other agents. This keeps agent identity without an index-based agent encoding, which would break permutation invariance. Agents beyond a distance threshold are masked out.
Joint latent intent. Each agent has a latent code; all codes are inferred jointly from all agents’ futures and decoded jointly, so one agent’s intent shapes others’ trajectories. The decoder is autoregressive.
Diverse sampling. A DLow-style trajectory sampler maps the context to K sets of latent codes for diverse, plausible samples.

Results

ETH/UCY (K=20): average ADE/FDE 0.23/0.39 m, the best FDE in the table (Trajectron++ 0.19/0.41, STAR 0.26/0.53).
nuScenes vehicles: ADE/FDE 1.86/3.89 m with 5 samples and 1.45/2.86 m with 10, ahead of Trajectron++ (ADE 1.88 and 1.51) and DLow-AF (FDE10 3.58).
Ablations on ETH/UCY: the best separate social+temporal combination (Transformer+Transformer) reaches 0.26/0.47; replacing agent-aware attention with standard attention or an agent encoding gives 0.27/0.47 and 0.27/0.46.

Limitations

The paper states none. Trajectron++ keeps a lower average ADE on ETH/UCY (0.19 vs 0.23 m). The autoregressive decoder trades training speed for fewer compounding errors; training takes about a day per ETH/UCY split and three days on nuScenes on one RTX 2080 Ti. Evaluation covers benchmark datasets only, predicting 2D positions.

Figures

From Figure 1: Standard models handle multi-agent trajectories with separate temporal and social models; AgentFormer ... — From Figure 1: Standard models handle multi-agent trajectories with separate temporal and social models; AgentFormer models both dimensions jointly on one flattened sequence while keeping time and agent information.

From Figure 2: Agent-aware attention. A mask M computes each attention weight with "self" queries and keys when the query and key belong to the same agent, and with "other" queries and keys when they do not.

From Figure 3: The CVAE prediction framework: a past encoder, prior, future encoder, autoregressive future decoder an... — From Figure 3: The CVAE prediction framework: a past encoder, prior, future encoder, autoregressive future decoder and trajectory sampler, all built on AgentFormer encoders and decoders.

From Figure 4: Three sampled multi-agent futures (green) for one scene show following and collision avoidance; the at... — From Figure 4: Three sampled multi-agent futures (green) for one scene show following and collision avoidance; the attention map for sample 1 shows the model focusing on the turning points of nearby agents and on the target's own past.

Abstract

Predicting accurate future trajectories of multiple agents is essential for autonomous systems, but is challenging due to the complex agent interaction and the uncertainty in each agent's future behavior. Forecasting multi-agent trajectories requires modeling two key dimensions: (1) time dimension, where we model the influence of past agent states over future states; (2) social dimension, where we model how the state of each agent affects others. Most prior methods model these two dimensions separately, e.g., first using a temporal model to summarize features over time for each agent independently and then modeling the interaction of the summarized features with a social model. This approach is suboptimal since independent feature encoding over either the time or social dimension can result in a loss of information. Instead, we would prefer a method that allows an agent's state at one time to directly affect another agent's state at a future time. To this end, we propose a new Transformer, AgentFormer, that jointly models the time and social dimensions. The model leverages a sequence representation of multi-agent trajectories by flattening trajectory features across time and agents. Since standard attention operations disregard the agent identity of each element in the sequence, AgentFormer uses a novel agent-aware attention mechanism that preserves agent identities by attending to elements of the same agent differently than elements of other agents. Based on AgentFormer, we propose a stochastic multi-agent trajectory prediction model that can attend to features of any agent at any previous timestep when inferring an agent's future position. The latent intent of all agents is also jointly modeled, allowing the stochasticity in one agent's behavior to affect other agents. Our method substantially improves the state of the art on well-established pedestrian and autonomous driving datasets.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong