Social Attention: Modeling Attention in Human Crowds

Anirudh Vemula, Katharina Muelling, Jean Oh

Seminal October 2017

TL;DR A pedestrian trajectory predictor that learns soft attention over everyone in the crowd instead of only nearby neighbours, built as a structural RNN over a spatio-temporal graph. On the ETH and UCY datasets it cuts average displacement error from 0.37 m (Social LSTM) to 0.30 m.

What they do

A robot moving through a crowd has to predict where the people around it will walk. Earlier models such as Social LSTM assume only people in a local neighbourhood affect each other. The authors argue that velocity and heading also matter: someone running at you from the far end of a hallway matters more than a nearby person walking alongside you. Social Attention therefore predicts every pedestrian’s trajectory jointly, with a learned soft attention over all other people in the scene.

How it works

Spatio-temporal graph. Nodes are people. Spatial edges connect every pair of people at the same time-step and carry their relative position. Temporal edges connect a person to themselves at the next step and carry their displacement.
Structural RNN. Each factor of the graph is an LSTM: shared spatial edgeRNNs, temporal edgeRNNs and nodeRNNs. Sharing weights keeps the parameter count independent of crowd size.
Attention. For each person, scaled dot-product attention scores their temporal-edge hidden state against the hidden states of all their spatial edges. The weighted sum feeds the nodeRNN, which outputs a bivariate Gaussian over the next position.
Training and inference. Trained end to end on negative log-likelihood. At test time predicted positions are fed back in.

Results

ETH and UCY: 5 crowd sets, 1536 pedestrians, evaluated leave-one-out. The model observes 8 steps (3.2 s) and predicts 12 (4.8 s).
Average displacement error is 0.30 m, against 0.37 m for Social LSTM and 0.39 m for an independent LSTM. Final displacement error is 2.59 m, 3.32 m and 3.84 m respectively. Social Attention beats Social LSTM on every crowd set.
The biggest gain is on ETH-Hotel (0.29 vs 0.42 m ADE), which has many stationary pedestrians and people approaching each other head-on.
GPU inference runs in real time at 10 Hz.

Limitations

Evaluated only on recorded datasets, with no robot in the loop and no static obstacles. The independent LSTM is sometimes slightly better in sparse crowds. The attention can be wrong: it sometimes gives high weight to distant or stationary people and ignores nearby ones, and the authors leave the cause to future work.

Figures

From Figure 1: A person navigating a crowd attends to only a subset of the surrounding people at each time-step; the ... — From Figure 1: A person navigating a crowd attends to only a subset of the surrounding people at each time-step; the model learns this attention to predict everyone's trajectory.

From Figure 2: An example spatio-temporal graph of three people, the graph unrolled over two time-steps, and its fact... — From Figure 2: An example spatio-temporal graph of three people, the graph unrolled over two time-steps, and its factor graph.

From Figure 3: Architecture of the edgeRNN (left), the attention module (middle) and the nodeRNN (right).

From Figure 4: Predicted attention weights for the pedestrian in red. The circle around each other pedestrian's curre... — From Figure 4: Predicted attention weights for the pedestrian in red. The circle around each other pedestrian's current position has a radius proportional to its attention weight; (a)-(c) are successes and (d)-(f) failures.

Abstract

Robots that navigate through human crowds need to be able to plan safe, efficient, and human predictable trajectories. This is a particularly challenging problem as it requires the robot to predict future human trajectories within a crowd where everyone implicitly cooperates with each other to avoid collisions. Previous approaches to human trajectory prediction have modeled the interactions between humans as a function of proximity. However, that is not necessarily true as some people in our immediate vicinity moving in the same direction might not be as important as other people that are further away, but that might collide with us in the future. In this work, we propose Social Attention, a novel trajectory prediction model that captures the relative importance of each person when navigating in the crowd, irrespective of their proximity. We demonstrate the performance of our method against a state-of-the-art approach on two publicly available crowd datasets and analyze the trained attention model to gain a better understanding of which surrounding agents humans attend to, when navigating in a crowd.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong