Future Person Localization in First-Person Videos

Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani, Yoichi Sato

Seminal November 2017

TL;DR Predict where a person seen by a wearable camera will be one second later, from their past locations, scale and pose plus the wearer's ego-motion. On a new 4.5-hour first-person walking dataset the error at one second is 77 px on average, against 98-118 px for constant-velocity, nearest-neighbor and Social LSTM baselines.

What they do

The paper introduces a new task, future person localization: from a short wearable-camera clip of a person, predict where they will appear in the next frames, as a step toward assistive navigation in crowds. It uses three cues specific to first-person video: the wearer’s ego-motion, the target’s scale (a cue to perspective) and the target’s pose.

How it works

Inputs. From OpenPose keypoints: location (hip midpoint), scale (neck-to-hip distance) and a 36-D normalized pose. Ego-motion is a 6-D rotation and translation per frame from an unsupervised estimator.
Network. Three streams of 1D temporal convolutions are concatenated and decoded by 1D deconvolutions into future location-scale offsets. Trained with an MSE loss.
Setup. One second in, one second out at 10 fps. The metric is final displacement error (FDE) in pixels on 1280x960 frames.
Data. The new First-Person Locomotion dataset: about 4.5 h of chest-camera walking video, more than 5,000 people, about 50,000 samples.

Results

Average FDE is 77.26 px, against 107.15 for constant velocity, 98.38 for nearest neighbor and 118.10 for a modified Social LSTM. People walking toward the camera are hardest: 109.03 here, 166-179 for the baselines.
73% of predictions are within 100 px (about 10 degrees). Assuming a 60 cm hip-to-neck length, the average error is about 60 cm.
Ablation: locations alone give 88.16. Each cue helps, and all three together give 77.26.
On the First-Person Social Interaction dataset it gets 125.42, against 162.02-213.59 for the baselines.

Limitations

All methods fail when the wearer moves suddenly and unexpectedly. The authors suggest also predicting the wearer’s own motion. The ego-motion estimator broke down when wearers turned their heads often, so the second dataset used optical flow instead, and there ego-motion barely helped (127.40 to 127.28 px). Predictions are 2D image locations, at most two seconds ahead.

Figures

From Figure 1: Given a first-person video of a target person, the network predicts where that person will be in futur... — From Figure 1: Given a first-person video of a target person, the network predicts where that person will be in future frames from their pose and scale and the camera wearer's ego-motion.

From Figure 2: Past frames are the input and the target's locations in the following frames are predicted, using the ... — From Figure 2: Past frames are the input and the target's locations in the following frames are predicted, using the target's locations and scales, the camera wearer's ego-motion and the target's poses as cues.

From Figure 3: Network architecture. Location-scale, ego-motion and pose streams of 1D convolutions are concatenated ... — From Figure 3: Network architecture. Location-scale, ego-motion and pose streams of 1D convolutions are concatenated channel-wise and decoded by deconvolutions into future offsets.

From Figure 5: Example predictions. Past observations on the left, future frames on the right with ground truth (red)... — From Figure 5: Example predictions. Past observations on the left, future frames on the right with ground truth (red), the method (green), NNeighbor (cyan) and Social LSTM (yellow).

Abstract

We present a new task that predicts future locations of people observed in first-person videos. Consider a first-person video stream continuously recorded by a wearable camera. Given a short clip of a person that is extracted from the complete stream, we aim to predict that person's location in future frames. To facilitate this future person localization ability, we make the following three key observations: a) First-person videos typically involve significant ego-motion which greatly affects the location of the target person in future frames; b) Scales of the target person act as a salient cue to estimate a perspective effect in first-person videos; c) First-person videos often capture people up-close, making it easier to leverage target poses (e.g., where they look) for predicting their future locations. We incorporate these three observations into a prediction framework with a multi-stream convolution-deconvolution architecture. Experimental results reveal our method to be effective on our new dataset as well as on a public social interaction dataset.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong