DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B. Choy, Philip H. S. Torr, Manmohan Chandraker

Seminal April 2017

TL;DR A CVAE samples many possible futures for every agent in a scene, and an RNN ranks them by accumulated future reward, using the scene and the other agents, then iteratively refines them. On KITTI the best of the top 5 of 50 samples is off by 2.06 m at 4 s, against 3.86 m for a single RNN encoder-decoder prediction.

What they do

DESIRE predicts where every agent in a scene will be up to 4 s ahead, from 2 s of past trajectory and a scene image or top-down map. Because a car at an intersection may turn either way, it samples many hypotheses, then ranks and refines them, in one end-to-end network.

How it works

Sample generation. A GRU encodes the past trajectory. A CVAE latent, drawn from N(0, I) at test time, masks that encoding, and a GRU decoder turns each draw into one hypothesis (K = 50).
IOC-style ranking. A second RNN runs along each hypothesis and outputs a reward at every time step; the score is the reward summed over the whole future, so no goal has to be given.
Refinement. The same RNN regresses a displacement for each hypothesis; the corrected samples are fed back and re-scored (up to 4 iterations).
Scene context fusion. Each RNN step sees the sample’s velocity, CNN scene features pooled at its position, and the other agents’ hidden states pooled on a log-polar grid.

Results

Errors are oracle errors at 4 s: the best of the top 10% (5 of 50) ranked samples.

KITTI (about 2,500 examples): 2.06 m and a 0.41 miss rate (> 1 m) for DESIRE-SI with 4 iterations, against 3.86 m / 0.62 for RNN ED, 3.07 m / 0.59 for unranked CVAE samples and 5.62 m for linear regression.
Stanford Drone Dataset (about 16,000 examples): 5.33 px at 1/5 resolution, against 8.80 for RNN ED-SI and 9.54 for RNN ED.
Refinement lowers the error from 2.72 to 2.06 m on KITTI and from 7.61 to 5.33 px on SDD. Interaction features help on SDD (5.62 to 5.33 px) but barely on KITTI (2.07 to 2.06 m), which has few interacting cars.
The top 2 samples already match the best baselines on both datasets.

Limitations

The single top-ranked prediction is worse than direct regression (4.45 vs 3.86 m on KITTI, 10.65 vs 8.80 px on SDD), so the gains depend on using several samples. Past trajectories come from annotated tracklets, KITTI maps keep only static scene elements, and SDD is evaluated only in image pixels.

Figures

From Figure 1: A driving scenario in which the white van may steer left or right while avoiding other agents. DESIRE'... — From Figure 1: A driving scenario in which the white van may steer left or right while avoiding other agents. DESIRE's diverse future predictions (blue paths) are drawn on the camera image and on the top-down scene feature map.

From Figure 2: Overview. A CVAE-based RNN encoder-decoder generates multiple prediction samples (Sample Generation Mo... — From Figure 2: Overview. A CVAE-based RNN encoder-decoder generates multiple prediction samples (Sample Generation Module). An RNN then gives each sample a reward at every time step and regresses a displacement for it (Ranking and Refinement Module), refined by iterative feedback. The final prediction is the sample with the highest accumulated future reward; the aquamarine paths are used only in training.

From Figure 4: Oracle maximum L2 error on KITTI against the fraction of top-ranked samples used. Single-prediction ba... — From Figure 4: Oracle maximum L2 error on KITTI against the fraction of top-ranked samples used. Single-prediction baselines are horizontal lines (green Linear, black RNN ED, cyan RNN ED-SI); blue is unranked CVAE samples; magenta and red are DESIRE-S and DESIRE-SI, dotted without refinement and solid after four iterations.

From Figure 6: Past (red) and ground-truth future (blue) in (a), baselines in (b) and DESIRE in (c), on KITTI (top th... — From Figure 6: Past (red) and ground-truth future (blue) in (a), baselines in (b) and DESIRE in (c), on KITTI (top three rows) and the Stanford Drone Dataset (bottom three). In rows 1 and 2, RNN ED-SI turns only after it nears non-drivable area, while DESIRE anticipates the turn. In row 4 a cyclist turning right could also go around the roundabout; DESIRE-SI's top prediction takes that alternative and its top 10 cover the ground truth.

Abstract

We introduce a Deep Stochastic IOC RNN Encoder-decoder framework, DESIRE, for the task of future predictions of multiple interacting agents in dynamic scenes. DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i.e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents. DESIRE achieves these in a single end-to-end trainable neural network model, while being computationally efficient. The model first obtains a diverse set of hypothetical future prediction samples employing a conditional variational auto-encoder, which are ranked and refined by the following RNN scoring-regression module. Samples are scored by accounting for accumulated future rewards, which enables better long-term strategic decisions similar to IOC frameworks. An RNN scene context fusion module jointly captures past motion histories, the semantic scene context and interactions among multiple agents. A feedback mechanism iterates over the ranking and refinement to further boost the prediction accuracy. We evaluate our model on two publicly available datasets: KITTI and Stanford Drone Dataset. Our experiments show that the proposed model significantly improves the prediction accuracy compared to other baseline methods.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong