On human motion prediction using recurrent neural networks
TL;DR On Human3.6M, simply repeating the last observed pose beats the deep RNN motion predictors of the time (ERD, LSTM-3LR, SRNN) in the short term. A single-GRU seq2seq model that feeds back its own predictions and predicts velocities through a residual connection beats both, e.g. walking error at 80 ms drops from 0.81 (SRNN) to 0.28.
What they do
Deep RNN models of human motion (ERD, LSTM-3LR, SRNN) were judged on short-term angle error and on long-term plausibility. The authors show that on Human3.6M a zero-velocity baseline, which just repeats the last observed pose, beats all three in the short term, because they jump away from the conditioning poses in the first predicted frame. They then propose a simpler RNN that removes the jump.
How it works
- Seq2seq with shared weights. A single GRU (1024 units) encodes 2 s of observed motion and decodes the next 400 ms (or 1 s); encoder and decoder share weights, and a linear layer maps to the 54 joint angles. No spatial encoder, no stacked layers.
- Sampling-based loss. During training the decoder always takes its own previous prediction as input, as at test time. This replaces the hard-to-tune noise schedules of prior work.
- Residual connection. Each decoder step adds its output to its input, so the network models angle velocities while the loss stays in angle space.
- Multi-action training. One model on all 15 Human3.6M actions, optionally with a one-hot action label; it trains in a few hours.
Results
- Zero-velocity beats ERD, LSTM-3LR and SRNN at every horizon on walking, eating, smoking and discussion (walking at 80 ms: 0.39 vs 0.77-0.93 mean angle error).
- The sampling-based loss alone still starts badly (walking 0.92 at 80 ms); adding the residual connection brings it to 0.34.
- Training on all actions helps most: walking at 400 ms goes from 1.09 (single-action residual) to 0.81 (multi-action, supervised), against 1.30 for SRNN. Action labels help only by a small margin.
Limitations
Aperiodic actions stay hard and zero-velocity often still wins there: on sitting down it scores 1.05 at 400 ms against 1.62 for the model, and on discussion 1.04 against 1.09. The best numbers do not give the best long-term motion; the authors argue the deterministic angle loss is unsuited to long horizons and that one RNN struggles to do both. Errors are averaged over eight test sequences from a single subject.
Figures
Abstract
Human motion modelling is a classical problem at the intersection of graphics and computer vision, with applications spanning human-computer interaction, motion synthesis, and motion prediction for virtual and augmented reality. Following the success of deep learning methods in several computer vision tasks, recent work has focused on using deep recurrent neural networks (RNNs) to model human motion, with the goal of learning time-dependent representations that perform tasks such as short-term motion prediction and long-term human motion synthesis. We examine recent work, with a focus on the evaluation methodologies commonly used in the literature, and show that, surprisingly, state-of-the-art performance can be achieved by a simple baseline that does not attempt to model motion at all. We investigate this result, and analyze recent RNN methods by looking at the architectures, loss functions, and training procedures used in state-of-the-art approaches. We propose three changes to the standard RNN models typically used for human motion, which result in a simple and scalable RNN architecture that obtains state-of-the-art performance on human motion prediction.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.