Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction

Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, Christian Claudel

Seminal February 2020

TL;DR Social-STGCNN treats the pedestrians in a scene as a spatio-temporal graph, applies graph convolutions, then predicts all future trajectories in one pass with a temporal CNN. On ETH/UCY it cuts average final displacement error from 0.94 to 0.75 using 7.6K parameters and 0.002 s per inference step.

What they do

Earlier pedestrian predictors model each person with a recurrent network and combine those hidden states with pooling or attention layers. Social-STGCNN instead treats the scene as a graph from the start: pedestrians are nodes, edges are weighted by how close people are, and convolutions replace both the recurrence and the aggregation. It observes 8 frames (3.2 s) and outputs a bivariate Gaussian for each pedestrian over the next 12 frames (4.8 s).

How it works

Graph. At each time step, each node holds a pedestrian’s observed position. Edge weights come from a kernel, the inverse distance between two pedestrians, set to 0 when they are at the same spot. The adjacency matrices are symmetrically normalized.
ST-GCNN. Spatio-temporal graph convolutions turn the observed graph into an embedding.
TXP-CNN. A stack of residual CNNs treats time as feature channels and extrapolates the embedding to the future horizon, so the whole sequence comes out in one shot. Training minimizes the negative log-likelihood.
The best setup has 1 ST-GCNN layer and 5 TXP-CNN layers; more graph layers made results worse.

Results

ETH/UCY, best of 20 samples: average ADE/FDE of 0.44/0.75, against 0.45/0.94 for SR-LSTM-2, the previous best FDE.
7.6K parameters and 0.002 s per inference step on a GTX 1080 Ti. S-GAN-P has 46.3K parameters and takes 0.0968 s (48x slower); SR-LSTM-2 has 64.9K (8.5x more).
Kernel ablation: the inverse-distance kernel gives 0.44/0.75. L2 distance gives 0.48/0.84, and an all-ones adjacency gives 0.49/0.79.
With 20% of the training data it beats the previous best FDE and Social-GAN’s ADE; Social-GAN improves little with more data.

Limitations

The model sees only pedestrian positions, with no scene context. Some sampled trajectories collide or diverge, and the supplementary results show cases where the averaged trajectory fails. Graph convolutions did not scale in depth. The authors leave other agents, such as bicycles and cars, to future work.

Figures

From Figure 1: A spatio-temporal graph represents the observed trajectories and the social interactions between pedes... — From Figure 1: A spatio-temporal graph represents the observed trajectories and the social interactions between pedestrians; the model predicts all future trajectories in a single pass.

From Figure 2: The model. A spatio-temporal graph is built from T observed frames, ST-GCNN layers turn it into a spat... — From Figure 2: The model. A spatio-temporal graph is built from T observed frames, ST-GCNN layers turn it into a spatio-temporal embedding, and TXP-CNN layers predict the future trajectories as a distribution.

From Figure 4: Ground truth (left) and samples from the model when pedestrians meet or merge. Samples change directio... — From Figure 4: Ground truth (left) and samples from the model when pedestrians meet or merge. Samples change direction or speed to avoid collisions; the last column and the bottom row show undesired and failed samples.

From Figure 5: ADE and FDE of Social-STGCNN and Social-GAN trained on random subsets of the training data, with the p... — From Figure 5: ADE and FDE of Social-STGCNN and Social-GAN trained on random subsets of the training data, with the previous state of the art as reference lines.

Abstract

Better machine understanding of pedestrian behaviors enables faster progress in modeling interactions between agents such as autonomous vehicles and humans. Pedestrian trajectories are not only influenced by the pedestrian itself but also by interaction with surrounding objects. Previous methods modeled these interactions by using a variety of aggregation methods that integrate different learned pedestrians states. We propose the Social Spatio-Temporal Graph Convolutional Neural Network (Social-STGCNN), which substitutes the need of aggregation methods by modeling the interactions as a graph. Our results show an improvement over the state of art by 20% on the Final Displacement Error (FDE) and an improvement on the Average Displacement Error (ADE) with 8.5 times less parameters and up to 48 times faster inference speed than previously reported methods. In addition, our model is data efficient, and exceeds previous state of the art on the ADE metric with only 20% of the training data. We propose a kernel function to embed the social interactions between pedestrians within the adjacency matrix. Through qualitative analysis, we show that our model inherited social behaviors that can be expected between pedestrians trajectories. Code is available at https://github.com/abduallahmohamed/Social-STGCNN.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong