Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Peeking into the Future: Predicting Future Person Activities and Locations in Videos

Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander Hauptmann, Li Fei-Fei

Seminal February 2019

TL;DR Next predicts where a pedestrian will walk over the next 4.8 s and which of 29 activities they will be doing, from features of the person's appearance, pose and surroundings plus an auxiliary task of predicting where the activity happens. On ActEV/VIRAT it lowers single-model trajectory error to 17.99 px ADE, down from 23.10 px for Social LSTM.

What they do

Most trajectory predictors treat a pedestrian as a point. Next encodes the person’s appearance, body motion and surroundings. From these it predicts the person’s future path and their future activity, one of 29 NIST classes such as “load” or “open_door”. The model watches 3.2 s (8 frames) and predicts the next 4.8 s (12 frames).

How it works

  • Person behavior module. Detector features from each person box plus 17 body keypoints, each encoded by its own LSTM.
  • Person interaction module. Scene-segmentation features pooled around the person, plus log-scaled distance and size ratios to every other person and object, with their classes.
  • Trajectory generator. An LSTM decoder outputs xy-coordinates and applies focal attention over all the encoders’ states.
  • Activity prediction. A multi-label activity classifier plus an auxiliary task that predicts where the activity happens on a two-scale “Manhattan Grid”: classify the cell, then regress the offset from its center.

Results

  • ActEV/VIRAT, single model: 17.99 / 37.24 px ADE/FDE, against 23.10 / 44.27 for Social LSTM and 30.48 / 62.17 for SGAN-V. With 20 outputs: 16.00 / 32.99, against 21.16 / 38.05 for SGAN-V-20.
  • Activity prediction reaches 0.192 mAP. Outputting the activity detected at the last observed frame gives 0.155.
  • Ablations: removing both auxiliary tasks raises ADE/FDE to 20.37 / 42.79. Without the location task, mAP drops to 0.152.
  • ETH & UCY (path only): 0.46 / 1.00 m average with 20 outputs, against 0.54 / 1.15 for SoPhie.

Limitations

The ActEV results use ground-truth boxes during observation; with detection and tracking inputs, ADE rises to 34.32 px. The model only predicts the predefined activities, a few seconds ahead, and misses subtle interactions such as two people stopping to talk. It has not been tested across different populations.

Figures

From Figure 1: The goal is to predict a person's future path and activity jointly. Depending on the future activity, ...
From Figure 1: The goal is to predict a person's future path and activity jointly. Depending on the future activity, the person at top right may take the yellow path to load the car or the green path for an object transfer.
From Figure 2: Model overview. A person behavior module and a person interaction module (person-scene and person-obje...
From Figure 2: Model overview. A person behavior module and a person interaction module (person-scene and person-object relations) encode the observed frames into a feature tensor, which feeds a trajectory generator and an activity prediction head.
From Figure 6: Predictions of LSTM, Social-LSTM, SGAN-V-20 and Next-20 on the same scene. Yellow is the observed path...
From Figure 6: Predictions of LSTM, Social-LSTM, SGAN-V-20 and Next-20 on the same scene. Yellow is the observed path, green the ground-truth future and blue heatmaps the predictions; Next also labels each person's future activity.
Abstract

Deciphering human behaviors to predict their future paths/trajectories and what they would do from videos is important in many applications. Motivated by this idea, this paper studies predicting a pedestrian's future path jointly with future activities. We propose an end-to-end, multi-task learning system utilizing rich visual features about human behavioral information and interaction with their surroundings. To facilitate the training, the network is learned with an auxiliary task of predicting future location in which the activity will happen. Experimental results demonstrate our state-of-the-art performance over two public benchmarks on future trajectory prediction. Moreover, our method is able to produce meaningful future activity prediction in addition to the path. The result provides the first empirical evidence that joint modeling of paths and activities benefits future path prediction.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.