Learning Trajectory Dependencies for Human Motion Prediction

Wei Mao, Miaomiao Liu, Mathieu Salzmann, Hongdong Li

Seminal August 2019

TL;DR A feed-forward network predicts future human poses by encoding each joint's trajectory as DCT coefficients and relating joints through a graph convolutional network that learns its own connectivity. Trained on 3D joint positions, it lowers average 400 ms error on Human3.6M from 80.2 mm (convSeq2Seq) to 61.3 mm.

What they do

Given 10 observed poses (400 ms), the task is to forecast the next 10 (400 ms) or 25 (1 s). RNN predictors accumulate errors and often jump between the last observed and the first predicted frame. This paper uses a feed-forward network that works in trajectory space instead of pose space: each joint’s trajectory becomes a set of DCT coefficients, and a graph convolutional network with learned connectivity models how joints depend on each other.

How it works

DCT encoding. The observed sequence is padded by repeating the last pose and converted to per-joint DCT coefficients. The network predicts the residual to the coefficients of the full observed-plus-future sequence, and the inverse DCT gives the poses. Keeping only low frequencies (15 of 35 coefficients for short-term 3D prediction) keeps trajectories smooth.
Learned graph. Joints form a fully connected graph, and every graph-convolution layer has its own trainable adjacency matrix, so dependencies are not limited to the kinematic tree.
Network and loss. 12 residual blocks of two graph-convolution layers plus an input and an output layer, about 2.6M parameters. Training uses an ℓ1 loss on joint angles or MPJPE on 3D positions, over observed and future frames.

Results

Human3.6M, trained on 3D: 12.1 / 25.0 / 51.0 / 61.3 mm at 80 / 160 / 320 / 400 ms, versus 19.6 / 37.8 / 68.1 / 80.2 mm for convSeq2Seq trained on 3D. Long-term (4 actions): 50.4 mm at 560 ms and 71.0 mm at 1 s, versus 62.9 and 85.4 mm.
3DPW, 3D: 35.6 mm at 200 ms and 117.8 mm at 1 s, versus 71.6 and 187.5 mm for convSeq2Seq.
Ablations (3D, 400 ms, 4 actions): 38.4 mm with learned connectivity, 42.8 mm with a fully connected network and 71.0 mm with a kinematic-tree graph. Removing padding or residual connections raises the error to about 50 mm.
On Phoning, the angle errors of three methods are similar while their 3D errors differ widely. The authors argue that angle-space evaluation is unreliable because different angles can describe the same pose.

Limitations

The paper has no limitations section. Its tables show it is not best everywhere: on Human3.6M at 1 s its average angle error is 1.27 versus 1.09 for adversarially trained AGED, on Phoning it is 1.48 versus 0.68 at 400 ms, and on CMU-Mocap its 1 s 3D error is 96.5 mm versus 84.6 mm for convSeq2Seq. Global rotation and translation are removed, so only body pose relative to the root is predicted.

Figures

From Figure 1: Observed frames on the left, then short- and long-term predictions. From top to bottom: ground truth, ... — From Figure 1: Observed frames on the left, then short- and long-term predictions. From top to bottom: ground truth, Residual sup., convSeq2Seq, and this method on joint angles and on 3D coordinates.

From Figure 2: The DCT encodes the observed poses in trajectory space, 12 residual blocks of graph convolutional laye... — From Figure 2: The DCT encodes the observed poses in trajectory space, 12 residual blocks of graph convolutional layers with learned adjacency matrices predict residual DCT coefficients, and the inverse DCT turns them back into poses.

From Figure 4: Per-frame joint angle error (top) and 3D position error (bottom) on the Phoning action of Human3.6M. A... — From Figure 4: Per-frame joint angle error (top) and 3D position error (bottom) on the Phoning action of Human3.6M. All methods have a similar angle error, but Residual sup. has a much higher 3D error.

Abstract

Human motion prediction, i.e., forecasting future body poses given observed pose sequence, has typically been tackled with recurrent neural networks (RNNs). However, as evidenced by prior work, the resulted RNN models suffer from prediction errors accumulation, leading to undesired discontinuities in motion prediction. In this paper, we propose a simple feed-forward deep network for motion prediction, which takes into account both temporal smoothness and spatial dependencies among human body joints. In this context, we then propose to encode temporal information by working in trajectory space, instead of the traditionally-used pose space. This alleviates us from manually defining the range of temporal dependencies (or temporal convolutional filter size, as done in previous work). Moreover, spatial dependency of human pose is encoded by treating a human pose as a generic graph (rather than a human skeletal kinematic tree) formed by links between every pair of body joints. Instead of using a pre-defined graph structure, we design a new graph convolutional network to learn graph connectivity automatically. This allows the network to capture long range dependencies beyond that of human kinematic tree. We evaluate our approach on several standard benchmark datasets for motion prediction, including Human3.6M, the CMU motion capture dataset and 3DPW. Our experiments clearly demonstrate that the proposed approach achieves state of the art performance, and is applicable to both angle-based and position-based pose representations. The code is available at https://github.com/wei-mao-2019/LearnTrajDep

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong