Structural-RNN: Deep Learning on Spatio-Temporal Graphs
TL;DR S-RNN turns any spatio-temporal graph into a jointly trained mixture of RNNs, one per node and edge factor, shared across similar nodes. On the same graph as a spatio-temporal CRF it lifts CAD-120 object-affordance anticipation F1 from 36.7% to 80.7%, and it beats ERD and a 3-layer LSTM on short-term mocap forecasting.
What they do
Many problems are posed as spatio-temporal graphs (st-graphs): nodes for components such as body parts, humans or objects, edges for their interactions in space and over time. Structural-RNN (S-RNN) is a generic recipe that casts any st-graph as a feedforward, fully differentiable mixture of RNNs trained as one network. It is tested on human motion forecasting, human-object interaction and driver maneuver anticipation.
How it works
- Factorize. The st-graph becomes a factor graph with a factor per node and per edge. Semantically similar nodes (e.g. all objects) share one node factor, which groups edges into shared edge factors; spatial and temporal edges stay separate.
- One RNN per factor. Edge factors become edgeRNNs, node factors nodeRNNs. Each nodeRNN concatenates its node features with the outputs of its neighboring edgeRNNs and predicts the node label.
- Any number of objects. Features of several edges of one type are summed before the shared edgeRNN, so a fixed architecture handles varying graph sizes.
Results
- H3.6m forecasting (st-graph over spine, arms, legs): lowest angle error at 80-320 ms on all four activities, e.g. 1.67 vs 2.25 (LSTM-3LR) and 2.67 (ERD) on discussion at 80 ms. Without edgeRNNs the skeleton freezes to a mean pose.
- CAD-120, same st-graph as Koppula et al.’s CRF: affordance detection F1 81.5% to 88.7% (91.1% multi-task); sub-activity anticipation 37.9% to 62.3% (65.6% multi-task).
- Driving maneuvers (1100 miles), versus AIO-HMM on the same st-graph: all-maneuver precision 77.4% to 82.2%, recall 71.2% to 75.9%.
- A leg-nodeRNN cell fires when the leg moves forward. Swapping the left-leg nodeRNN between slow and fast models yields a new motion, a human jumping forward on one leg.
Limitations
The st-graph and node grouping are designed by hand, and the interaction and driving experiments reuse features from prior work. Unlike a CRF, S-RNN does not model the joint distribution over outputs. Long-horizon forecasts are not always best (eating at 1000 ms: ERD 2.41 vs 2.58), and the authors call long-term errors an unfair measure because human motion is stochastic.
Figures
Abstract
Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatio-temporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks (RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.