Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Human Motion Trajectory Prediction: A Survey

Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M. Kitani, Dariu M. Gavrila, Kai O. Arras

Seminal May 2019

TL;DR A survey of more than 200 human trajectory prediction methods for robots, cars and surveillance, sorted by modeling approach (physics, pattern, planning) and by the contextual cues they use. It also reviews metrics and datasets and finds that benchmarking lacks common practice; on ETH/UCY, reported errors are 0.19-0.4 m ADE at 4.8 s, against 0.53 m for linear velocity projection.

What they do

A survey of human motion trajectory prediction: ground-level 2D trajectories of pedestrians, plus cyclists and vehicles, for service robots, self-driving vehicles and surveillance. It covers more than 200 methods from robotics, intelligent vehicles and computer vision, proposes a two-axis taxonomy, reviews metrics and datasets, and asks three questions: is evaluation good enough, does the modeling approach still matter, and is prediction solved?

How it works

  • Modeling approach. Physics-based methods (sense-predict) forward-simulate explicit dynamics, with one model or several. Pattern-based methods (sense-learn-predict) learn the dynamics from data, sequentially or over whole trajectories. Planning-based methods (sense-reason-predict) reason about goals, with a given cost (forward planning) or a cost learned from observations (inverse planning).
  • Contextual cues, a separate axis: target-agent cues (motion state, articulated pose, semantic attributes), the dynamic environment (unaware, individual- or group-aware) and the static environment (unaware, obstacle-, map- or semantics-aware).
  • Evaluation. Geometric metrics (ADE, FDE, DTW, modified Hausdorff, LCS) and probabilistic ones (negative log-likelihood, predicted probability, minimum-of-K ADE/FDE); 12 datasets used by at least two independent teams, plus 4 recent ones.

Results

  • Publications rose sharply in 2018-2019, led by pattern-based methods.
  • Reported accuracy on ETH and UCY is 0.19-0.4 m ADE at 4.8 s, against 0.53 m for linear velocity projection; on the harder ATC dataset it is 1.4-2 m at 9 s.
  • Q1 and Q2 are answered no: benchmarking has no common practice (differing metric names and formulas, single arbitrary horizons, little robustness testing), and too few studies compare modeling approaches directly.
  • Q3: only specific use cases, such as basic emergency braking for intelligent vehicles, are judged solved.

Limitations

The comparison is qualitative: datasets, metrics and horizons differ across papers, so the accuracy figures are ranges reported by different works, not a common benchmark. Articulated motion, video-frame and action prediction are out of scope. Open problems the authors name: short, noisy trajectories with few interactions in the datasets, rare robustness testing, unclear transfer of learned models to new sites, and planning-based methods that need a given set of goals.

Figures

From Figure 2: A motion prediction system takes stimuli (trajectories, articulation and attributes, raw sensor data, ...
From Figure 2: A motion prediction system takes stimuli (trajectories, articulation and attributes, raw sensor data, an environment model) and outputs a single trajectory, a parametric distribution or a non-parametric distribution.
From Figure 3: The taxonomy splits methods by modeling approach (physics-, pattern- and planning-based, two subclasse...
From Figure 3: The taxonomy splits methods by modeling approach (physics-, pattern- and planning-based, two subclasses each) and by contextual cues of the target agent, the dynamic environment and the static environment.
From Figure 5: (a) Physics-based methods project the motion state with explicit dynamical models. (b) Pattern-based m...
From Figure 5: (a) Physics-based methods project the motion state with explicit dynamical models. (b) Pattern-based methods learn prototypical trajectories from observed behavior. (c) Planning-based methods reason about likely goals and compute paths to reach them.
From Figure 4: Publications per year in the reviewed literature, 1991 to 2019, color-coded by modeling approach.
From Figure 4: Publications per year in the reviewed literature, 1991 to 2019, color-coded by modeling approach.
Abstract

With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand and anticipate human behavior becomes increasingly important. Specifically, predicting future positions of dynamic agents and planning considering such predictions are key tasks for self-driving vehicles, service robots and advanced surveillance systems. This paper provides a survey of human motion trajectory prediction. We review, analyze and structure a large selection of work from different communities and propose a taxonomy that categorizes existing methods based on the motion modeling approach and level of contextual information used. We provide an overview of the existing datasets and performance metrics. We discuss limitations of the state of the art and outline directions for further research.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.