Human motion prediction
Predicting how and where people will move, from trajectories and gait to intent, for robots that work around people.
Start here: seminal papers
Recurrent Network Models for Human DynamicsSeminal
August 2015
ERD wraps stacked LSTMs in learned nonlinear encoder and decoder networks to generate motion capture, label body joints in video and forecast them 400 ms ahead. On Human3.6M its generated walking stays realistic long after a plain LSTM collapses to the mean pose, and it fixes left-right joint confusions of a per-frame CNN detector.
Structural-RNN: Deep Learning on Spatio-Temporal GraphsSeminal
November 2015
S-RNN turns any spatio-temporal graph into a jointly trained mixture of RNNs, one per node and edge factor, shared across similar nodes. On the same graph as a spatio-temporal CRF it lifts CAD-120 object-affordance anticipation F1 from 36.7% to 80.7%, and it beats ERD and a 3-layer LSTM on short-term mocap forecasting.
Socially Aware Motion Planning with Deep Reinforcement LearningSeminal
March 2017
SA-CADRL trains a deep-RL collision-avoidance value network with a small penalty for breaking a social norm, such as passing on the wrong side, and extends it to four agents. The right-handed policy picks the norm-respecting side in 83-98% of two-agent test cases, at about 20% more extra time to goal. It drove a robot fully autonomously among pedestrians at 1.2 m/s.
DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting AgentsSeminal
April 2017
A CVAE samples many possible futures for every agent in a scene, and an RNN ranks them by accumulated future reward, using the scene and the other agents, then iteratively refines them. On KITTI the best of the top 5 of 50 samples is off by 2.06 m at 4 s, against 3.86 m for a single RNN encoder-decoder prediction.
On human motion prediction using recurrent neural networks★ Must read
May 2017
On Human3.6M, simply repeating the last observed pose beats the deep RNN motion predictors of the time (ERD, LSTM-3LR, SRNN) in the short term. A single-GRU seq2seq model that feeds back its own predictions and predicts velocities through a residual connection beats both, e.g. walking error at 80 ms drops from 0.81 (SRNN) to 0.28.
Social Attention: Modeling Attention in Human CrowdsSeminal
October 2017
A pedestrian trajectory predictor that learns soft attention over everyone in the crowd instead of only nearby neighbours, built as a structural RNN over a spatio-temporal graph. On the ETH and UCY datasets it cuts average displacement error from 0.37 m (Social LSTM) to 0.30 m.
Future Person Localization in First-Person VideosSeminal
November 2017
Predict where a person seen by a wearable camera will be one second later, from their past locations, scale and pose plus the wearer's ego-motion. On a new 4.5-hour first-person walking dataset the error at one second is 77 px on average, against 98-118 px for constant-velocity, nearest-neighbor and Social LSTM baselines.
Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks★ Must read
March 2018
A GAN with LSTM encoder-decoders, a pooling layer over everyone in the scene and a best-of-k "variety" loss predicts several plausible future paths for each pedestrian. With 20 samples it lowers the 4.8 s average displacement error on ETH/UCY to 0.58 m, from 0.72 m for Social LSTM, and the pooled model runs 16x faster than Social LSTM.
Probabilistically Safe Robot Planning with Confidence-Based Human PredictionsSeminal
May 2018
A robot treats the rationality coefficient of its human-motion model as a measure of confidence in that model, infers it online, and plans around the resulting predictions with a 1% collision-probability limit. On 48 recorded walking trajectories, this kept a simulated quadcopter farther from people or got it to its goal sooner than fixed-confidence predictions when the person avoided an unmodeled obstacle or headed to an unmodeled goal.
SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical ConstraintsSeminal
June 2018
SoPhie predicts pedestrian paths with a GAN whose LSTM generator attends both to a top-view image of the scene and to the surrounding agents. It reaches an average displacement error of 0.54 m on ETH/UCY (Social GAN 0.58 m) and 16.27 px on the Stanford Drone Dataset (DESIRE 19.25 px).
Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real TimeSeminal
October 2018
A bidirectional LSTM, trained on IMU signals synthesized from motion capture, predicts full-body SMPL pose from just 6 body-worn IMUs in real time. It reaches 15.85° mean joint-angle error on TotalCapture, lower than the offline optimizer SIP (16.98°), and its live demo runs at about 29 fps.
Peeking into the Future: Predicting Future Person Activities and Locations in VideosSeminal
February 2019
Next predicts where a pedestrian will walk over the next 4.8 s and which of 29 activities they will be doing, from features of the person's appearance, pose and surroundings plus an auxiliary task of predicting where the activity happens. On ActEV/VIRAT it lowers single-model trajectory error to 17.99 px ADE, down from 23.10 px for Social LSTM.
AMASS: Archive of Motion Capture as Surface ShapesSeminal
April 2019
MoSh++ fits the SMPL body model, including hands and soft-tissue motion, to raw mocap markers and uses it to convert 15 mocap datasets into one archive, AMASS, with 42 hours, 346 subjects and 11,451 motions. On 4D scans recorded alongside mocap, it recovers body shape to 7.4 mm, versus 12.1 mm for the original MoSh.
Human Motion Trajectory Prediction: A SurveySeminal
May 2019
A survey of more than 200 human trajectory prediction methods for robots, cars and surveillance, sorted by modeling approach (physics, pattern, planning) and by the contextual cues they use. It also reviews metrics and datasets and finds that benchmarking lacks common practice; on ETH/UCY, reported errors are 0.19-0.4 m ADE at 4.8 s, against 0.53 m for linear velocity projection.
Learning Trajectory Dependencies for Human Motion PredictionSeminal
August 2019
A feed-forward network predicts future human poses by encoding each joint's trajectory as DCT coefficients and relating joints through a graph convolutional network that learns its own connectivity. Trained on 3D joint positions, it lowers average 400 ms error on Human3.6M from 80.2 mm (convSeq2Seq) to 61.3 mm.
Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data★ Must read
January 2020
A graph-structured generative model that forecasts many interacting agents by predicting control actions and integrating their dynamics, using semantic maps and optionally the robot's own plan. On ETH/UCY pedestrians its best-of-20 errors are 55-60% lower than the prior best.
Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory PredictionSeminal
February 2020
Social-STGCNN treats the pedestrians in a scene as a spatio-temporal graph, applies graph convolutions, then predicts all future trajectories in one pass with a temporal CNN. On ETH/UCY it cuts average final displacement error from 0.94 to 0.75 using 7.6K parameters and 0.002 s per inference step.
It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory PredictionSeminal
April 2020
PECNet first predicts where each pedestrian is heading with a VAE, then predicts the path to that sampled endpoint jointly with its neighbours. It lowers Stanford Drone error to 9.96 px ADE, 20.9% below the previous best, and ETH/UCY error by 40.8%.
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent ForecastingSeminal
March 2021
A Transformer that flattens all agents' trajectories into one sequence and uses agent-aware attention, so any agent's past state can directly inform another agent's future. It gets the best average FDE on ETH/UCY (0.39 m) and beats Trajectron++ on nuScenes (ADE10 1.45 m vs 1.51 m).
Ego4D: Around the World in 3,000 Hours of Egocentric VideoSeminal
October 2021
Ego4D is a 3,670-hour dataset of unscripted head-mounted video from 931 camera wearers in 74 locations across 9 countries, with dense narrations and five benchmarks covering the wearer's past, present and future. The baselines leave a lot of room for improvement, e.g. 2.45% Top-5 mAP on short-term object interaction anticipation.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.