Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data

Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, Marco Pavone

★ Must read January 2020

TL;DR A graph-structured generative model that forecasts many interacting agents by predicting control actions and integrating their dynamics, using semantic maps and optionally the robot's own plan. On ETH/UCY pedestrians its best-of-20 errors are 55-60% lower than the prior best.

What they do

Trajectron++ forecasts trajectories for a varying number of interacting agents of different types (pedestrians, cars). Most forecasters ignore agent dynamics, maps and the robot’s own plan. This model predicts control actions and integrates each agent’s dynamics, so every sampled trajectory is dynamically feasible. It also encodes semantic maps and can condition its forecasts on the ego-vehicle’s planned motion.

How it works

Scene graph. Agents are nodes. A directed edge is added when one agent lies within the other’s class-specific perception range. LSTMs encode each agent’s history and its summed same-class neighbors, and attention combines the edge types into one influence vector.
Extra inputs. A 4-layer CNN encodes a local map rotated to the agent’s heading. A bi-directional LSTM can encode the ego-vehicle’s future plan.
Multimodal output. A CVAE with a 25-way discrete latent picks a behavior mode. A GRU decoder outputs Gaussian controls: velocity for pedestrians (single integrators), acceleration and heading rate for vehicles (dynamically-extended unicycles).
Dynamics integration. The controls are integrated through each agent’s dynamics, linearized for the unicycle, to give position means and covariances.

Results

ETH/UCY pedestrians, best of 20 samples: 0.19 m ADE / 0.41 m FDE on average, versus 0.48 / 0.90 for MATF (55-60% lower).
KDE negative log-likelihood averages -1.14 with dynamics integration, versus -0.74 without it and 1.79 for the original Trajectron.
Map encoding cuts obstacle-colliding samples on ETH-Univ from 4.6% to 1.0% (21.5% to 4.9% for pedestrians near obstacles).
nuScenes vehicles: 1.14 m FDE at 3 s versus 1.23 m for SpAGNN. Conditioning on the ego plan lowers FDE at 4 s from 2.09 to 1.50 m and road-boundary violations from 7.6% to 4.2%.

Limitations

Vehicles use a unicycle rather than a bicycle model, because bicycle parameters are hard to estimate online. Edges come from a simple distance threshold. On nuScenes the model forecasts only pedestrians and cars, from annotated tracks. Baselines include detection and tracking, so the authors subtracted that error from the baselines’ reported numbers to compare. Dynamics integration alone slightly raises road-boundary violations (6.9% to 8.1% at 4 s).

Figures

From Figure 1: Pedestrians cross in front of a car that may go straight or turn right. Agents are graph nodes (white ... — From Figure 1: Pedestrians cross in front of a car that may go straight or turn right. Agents are graph nodes (white circles) joined by interaction edges (dashed lines). Arrows show possible future velocities, colored by high-level behavior mode.

From Figure 2: A nuScenes scene represented as a directed spatiotemporal graph, with agents (pedestrians, cars, the e... — From Figure 2: A nuScenes scene represented as a directed spatiotemporal graph, with agents (pedestrians, cars, the ego-vehicle R) as nodes and their interactions as edges.

From Figure 3: The base model on nuScenes undershoots the red car's turn, predicts it may end up in oncoming traffic,... — From Figure 3: The base model on nuScenes undershoots the red car's turn, predicts it may end up in oncoming traffic, and is overly confident.

From Figure 3: The same scene with dynamics integration and map encoding. Predictions are more accurate and nearly al... — From Figure 3: The same scene with dynamics integration and map encoding. Predictions are more accurate and nearly all probability mass stays on the correct side of the road.

Abstract

Reasoning about human motion is an important prerequisite to safe and socially-aware robotic navigation. As a result, multi-agent behavior prediction has become a core component of modern human-robot interactive systems, such as self-driving cars. While there exist many methods for trajectory forecasting, most do not enforce dynamic constraints and do not account for environmental information (e.g., maps). Towards this end, we present Trajectron++, a modular, graph-structured recurrent model that forecasts the trajectories of a general number of diverse agents while incorporating agent dynamics and heterogeneous data (e.g., semantic maps). Trajectron++ is designed to be tightly integrated with robotic planning and control frameworks; for example, it can produce predictions that are optionally conditioned on ego-agent motion plans. We demonstrate its performance on several challenging real-world trajectory forecasting datasets, outperforming a wide array of state-of-the-art deterministic and generative methods.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong