Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, S. Hamid Rezatofighi, Silvio Savarese

Seminal June 2018

TL;DR SoPhie predicts pedestrian paths with a GAN whose LSTM generator attends both to a top-view image of the scene and to the surrounding agents. It reaches an average displacement error of 0.54 m on ETH/UCY (Social GAN 0.58 m) and 16.27 px on the Stanford Drone Dataset (DESIRE 19.25 px).

What they do

SoPhie predicts the next 4.8 s of every pedestrian’s path from 3.2 s of history and one top-view image of the scene. Earlier methods use either the scene or the other agents, and many predict a single, averaged path. SoPhie combines both cues through two attention modules and samples multiple futures with a GAN.

How it works

  • Features. A VGG-19 (pretrained on ImageNet, fine-tuned on scene segmentation) encodes the image. An LSTM encodes each agent’s past track. For each target agent, the other agents’ encodings are sorted by distance to it rather than max-pooled, so each agent gets its own social feature (up to 32 agents, zero-padded).
  • Attention. Two soft attention modules, conditioned on the decoder LSTM’s hidden state, weight image regions (physical) and neighbours (social) into two context vectors.
  • GAN. The generator is an LSTM decoder fed both context vectors plus Gaussian noise. The discriminator is an LSTM that labels trajectories as real or generated. The loss is adversarial plus L2 (λ = 1).

Results

  • ETH/UCY (leave one scene out, meters): average ADE/FDE 0.54/1.15, versus 0.58/1.18 for Social GAN and 0.79/1.59 for a linear model. Either attention alone gives 0.57-0.58, no attention 0.61.
  • Stanford Drone Dataset (pixels): ADE/FDE 16.27/29.38, versus 19.25/34.05 for DESIRE and 25.72/51.8 for CAR-Net. On physically complex scenes ADE is 15.61, versus 29.29 for Social GAN and 24.32 for CAR-Net.
  • Near-collisions (closer than 0.10 m): 1.361% of pedestrians per frame on average, versus 1.717% for Social GAN and 0.189% in the ground truth.
  • Sampling the generator and filtering with the discriminator yields traversability maps of sidewalks and paths in drone scenes.

Limitations

The paper has no limitations section. Its tables show SoPhie worse than the linear baseline on Hotel (0.76 vs 0.39 m ADE), and with more near-collisions than Social GAN on Hotel and Univ. DESIRE’s numbers are interpolated from its 4.0 s result because its code was unavailable. Each scene uses one static image, and the agent count is capped at 32.

Figures

From Figure 1: SoPhie predicts socially and physically plausible trajectories by attending to all agents in the scene...
From Figure 1: SoPhie predicts socially and physically plausible trajectories by attending to all agents in the scene and to the scene itself.
From Figure 2: The architecture: (a) a feature extractor (CNN on the scene image, LSTM encoders on agent tracks), (b)...
From Figure 2: The architecture: (a) a feature extractor (CNN on the scene image, LSTM encoders on agent tracks), (b) physical and social attention for each agent, (c) an LSTM-based GAN with generator and discriminator.
From Figure 3: Social GAN versus SoPhie in three scenes where attention fixes Social GAN errors. Past paths are lines...
From Figure 3: Social GAN versus SoPhie in three scenes where attention fixes Social GAN errors. Past paths are lines and predictions are distributions; physical attention is shown in white on the image, and white boxes around agents show social attention relative to the blue agent.
Abstract

This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present SoPhie; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with a physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.