Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks

Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, Alexandre Alahi

★ Must read March 2018

TL;DR A GAN with LSTM encoder-decoders, a pooling layer over everyone in the scene and a best-of-k "variety" loss predicts several plausible future paths for each pedestrian. With 20 samples it lowers the 4.8 s average displacement error on ETH/UCY to 0.58 m, from 0.72 m for Social LSTM, and the pooled model runs 16x faster than Social LSTM.

What they do

Social GAN takes 3.2 s of observed paths for everyone in a scene and predicts all their futures jointly. There are many socially acceptable ways to avoid a collision, and models trained with an L2 loss learn their average. Social GAN instead samples several futures from a recurrent encoder-decoder trained adversarially.

How it works

Generator. A shared LSTM encodes each person’s past. The decoder LSTM starts from that encoding, a pooled social vector and noise z, and outputs coordinates directly.
Pooling module. Relative positions of all other people, concatenated with their hidden states, go through an MLP and a max-pool. Unlike Social LSTM’s local grid this covers the whole scene, and it runs once instead of at every time step.
Discriminator. A separate LSTM encoder classifies the observed path plus a real or generated future as real or fake.
Variety loss. The generator draws k samples per scene and only the one closest to the ground truth gets the L2 loss, so the samples spread over plausible futures.

Results

On ETH and UCY (1536 pedestrians, leave-one-out), the best of 20 samples without pooling averages 0.39 / 0.58 m ADE at 8 / 12 steps, against 0.45 / 0.72 m for S-LSTM and 0.43 / 0.70 m for LSTM. With pooling: 0.41 / 0.61 m.
Without the variety loss, one sample (0.49 / 0.74 m) is worse than LSTM and more samples do not help. Models trained with k = 100 are 33% more accurate on average.
Predicting 8 steps takes 0.12 s with pooling against 1.79 s for S-LSTM on a Tesla P100, a 16x speed-up.
In synthetic avoidance scenarios the pooled model gives way, overtakes or slows down; without pooling, one scenario ends in a collision.

Limitations

Pooling slightly worsens the error metrics; its benefit is shown only qualitatively. Accuracy is the best of N samples against the ground truth. The S-LSTM baseline, trained without its original synthetic pretraining, does not beat plain LSTM. Only person-to-person interactions are modeled, not the static scene, and pooling uses everyone’s true positions, which a real pedestrian’s field of view would not allow.

Figures

From Figure 1: Two pedestrians want to avoid each other, and there are many ways they can do it; given the same obser... — From Figure 1: Two pedestrians want to avoid each other, and there are many ways they can do it; given the same observed past, the method predicts multiple socially acceptable futures.

From Figure 2: System overview. The generator encodes each person's past trajectory, the pooling module turns all enc... — From Figure 2: System overview. The generator encodes each person's past trajectory, the pooling module turns all encodings into one pooled vector per person, and the decoder predicts the future from both; the discriminator classifies whole trajectories as real or fake.

From Figure 4: Effect of the variety loss on ETH, ZARA1 and ZARA2. Drawing N test samples from a model trained withou... — From Figure 4: Effect of the variety loss on ETH, ZARA1 and ZARA2. Drawing N test samples from a model trained without it (blue) barely lowers the error; models trained with it using N samples (red) are 29-34% more accurate at N = 100.

From Figure 5: The model without pooling (SGAN, top) and with pooling (SGAN-P, bottom) in four collision-avoidance sc... — From Figure 5: The model without pooling (SGAN, top) and with pooling (SGAN-P, bottom) in four collision-avoidance scenarios, showing the density and mean of 300 samples each; with pooling, the predicted trajectories avoid collisions.

Abstract

Understanding human motion behavior is critical for autonomous moving platforms (like self-driving cars and social robots) if they are to navigate human-centric environments. This is challenging because human motion is inherently multimodal: given a history of human motion paths, there are many socially plausible ways that people could move in the future. We tackle this problem by combining tools from sequence prediction and generative adversarial networks: a recurrent sequence-to-sequence model observes motion histories and predicts future behavior, using a novel pooling mechanism to aggregate information across people. We predict socially plausible futures by training adversarially against a recurrent discriminator, and encourage diverse predictions with a novel variety loss. Through experiments on several datasets we demonstrate that our approach outperforms prior work in terms of accuracy, variety, collision avoidance, and computational complexity.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong