It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction

Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, Adrien Gaidon

Seminal April 2020

TL;DR PECNet first predicts where each pedestrian is heading with a VAE, then predicts the path to that sampled endpoint jointly with its neighbours. It lowers Stanford Drone error to 9.96 px ADE, 20.9% below the previous best, and ETH/UCY error by 40.8%.

What they do

PECNet forecasts pedestrian trajectories in two steps: infer where each pedestrian will be at the end of the 4.8 s prediction window, then predict the path to a sampled endpoint jointly for everyone in the scene. The premise is that people head toward a destination, so the endpoint is easier to predict than the points in between.

How it works

Endpoint VAE. Past positions and the ground-truth endpoint are encoded into a Gaussian latent. At test time a latent sampled from N(0, σT I) is decoded with the past encoding into an endpoint.
Social pooling. Past and sampled-endpoint features of all pedestrians go through rounds of non-local attention, masked to neighbours close in space and overlapping in time. An MLP then outputs the remaining positions.
Training. End to end with KL, endpoint and trajectory losses. Downstream modules see sampled rather than ground-truth endpoints; using ground truth was about 8.8% worse.
Truncation trick. For small K, sample with σT = 1 and truncate; for large K, use σT > 1 without truncation. No retraining is needed.

Results

Stanford Drone, K = 20: 9.96 / 15.88 px ADE/FDE, against 12.58 / 22.07 for P2TIRL. Without social pooling or truncation: 10.56 ADE. It matches the previous best ADE with K = 10.
ETH/UCY, K = 20, no truncation: average 0.29 / 0.48 m, against 0.49 / 0.97 for CGNS. The gain is largest on HOTEL (74.2%) and smallest on ETH (12.9%).
Truncation at K = 1 lowers ADE from 22.85 to 17.29 (about 24.7%).
The error in predicting a future point grows up to about the 7th future position, then plateaus and drops slightly.

Limitations

The paper has no limitations section. As K grows, FDE approaches 0 but ADE stays at 6.49 px, so the remaining error is in the path to the endpoint. Social pooling adds only 0.33 ADE on Stanford Drone, and the truncation constant (c ≈ 1.2) is chosen empirically. Only pedestrian datasets are tested; there are no robot experiments.

Figures

From Figure 1: Trajectory prediction (top left) split into two steps: infer the distribution over endpoints (top righ... — From Figure 1: Trajectory prediction (top left) split into two steps: infer the distribution over endpoints (top right), then condition on sampled endpoints (bottom left) to jointly plan socially compliant trajectories for all agents (bottom right).

From Figure 2: PECNet architecture. Past history and the ground-truth endpoint train a VAE for multi-modal endpoint i... — From Figure 2: PECNet architecture. Past history and the ground-truth endpoint train a VAE for multi-modal endpoint inference; the sampled endpoints condition the social pooling and predictor networks. Red connections are used only during training.

From Figure 5: ADE and FDE (pixels) against the number of samples K, with and without the truncation trick, next to p... — From Figure 5: ADE and FDE (pixels) against the number of samples K, with and without the truncation trick, next to prior methods at the K they used.

From Figure 6: Multi-modal predictions from PECNet. White is the past 3.2 seconds; red and cyan are the predicted and... — From Figure 6: Multi-modal predictions from PECNet. White is the past 3.2 seconds; red and cyan are the predicted and ground-truth future over the next 4.8 seconds.

Abstract

Human trajectory forecasting with multiple socially interacting agents is of critical importance for autonomous navigation in human environments, e.g., for self-driving cars and social robots. In this work, we present Predicted Endpoint Conditioned Network (PECNet) for flexible human trajectory prediction. PECNet infers distant trajectory endpoints to assist in long-range multi-modal trajectory prediction. A novel non-local social pooling layer enables PECNet to infer diverse yet socially compliant trajectories. Additionally, we present a simple "truncation-trick" for improving few-shot multi-modal trajectory prediction performance. We show that PECNet improves state-of-the-art performance on the Stanford Drone trajectory prediction benchmark by ~20.9% and on the ETH/UCY benchmark by ~40.8%. Project homepage: https://karttikeya.github.io/publication/htf/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong