Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time

Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, Gerard Pons-Moll

Seminal October 2018

TL;DR A bidirectional LSTM, trained on IMU signals synthesized from motion capture, predicts full-body SMPL pose from just 6 body-worn IMUs in real time. It reaches 15.85° mean joint-angle error on TotalCapture, lower than the offline optimizer SIP (16.98°), and its live demo runs at about 29 fps.

What they do

Commercial inertial motion capture uses about 17 IMUs. The earlier 6-IMU method SIP needs offline optimization over the whole sequence, which takes minutes to hours. DIP (Deep Inertial Poser) learns a direct mapping from 6 IMUs (orientation plus acceleration) to SMPL body pose and runs in real time.

How it works

  • Synthetic data. Virtual sensors on the SMPL mesh are driven by AMASS mocap (2,703 minutes, 603 subjects). Orientations come from forward kinematics and accelerations from finite differences.
  • Network. Each frame gives 5 orientations (as rotation matrices) and 5 accelerations, normalized to the root sensor (60-D). Two stacked bidirectional LSTM layers output SMPL joint rotations with a Gaussian log-likelihood loss. An auxiliary head reconstructs the input acceleration so the network does not ignore it, and 20% of the inputs are dropped out.
  • Online use. The network is trained on full sequences but runs on a sliding window of 20 past and 5 future frames (85 ms latency).
  • Fine-tuning on DIP-IMU, a new real dataset: 10 subjects wearing 17 Xsens IMUs (about 90 minutes); its reference poses come from SIP run on all 17 sensors.

Results

  • TotalCapture (offline): 15.85° mean joint-angle error, against 16.98° for SIP and 22.18° for SOP. The online 20/5 window gives 15.88°.
  • DIP-IMU: 31.70° before fine-tuning (SIP: 24.00°) and 17.54° after. TotalCapture error rises only to 16.84°.
  • Acceleration matters: with the 20/5 window, removing the auxiliary loss raises error from 15.88° to 16.26°. Without acceleration inputs, error after fine-tuning is 19.45° on TotalCapture (16.84° with them).
  • The live Xsens-to-Unity demo runs at about 29 fps. On DIP-IMU, DIP rarely produces limb-torso interpenetrations, which SIP and SOP often do.

Limitations

Leg raises remain hard: orientations barely change, and the model struggles to use the noisy accelerations. The worst 5% of DIP-IMU poses average 43.68° error, and poses with the body parallel to the floor (push-ups) are challenging. A synthetic-to-real gap remains: on IMU data synthesized for TotalCapture, error is 2.71° lower than on real data. Global translation is not estimated.

Videos

Full video (YouTube)

Figures

From Figure 1: Live 3D pose reconstruction from six body-worn IMUs, including seated (occluded) and fast motions.
From Figure 1: Live 3D pose reconstruction from six body-worn IMUs, including seated (occluded) and fast motions.
From Figure 2: Pipeline. Mocap data drives virtual sensors on an SMPL mesh to synthesize IMU signals; a recurrent net...
From Figure 2: Pipeline. Mocap data drives virtual sensors on an SMPL mesh to synthesize IMU signals; a recurrent network maps IMU signals to SMPL pose parameters and recovers full-body pose from 6 sensors in real time.
From Figure 3: In training, two stacked bidirectional layers see the whole sequence (left); at runtime a sliding wind...
From Figure 3: In training, two stacked bidirectional layers see the whole sequence (left); at runtime a sliding window of past frames and a few future frames predicts the current pose (right).
From Figure 9: DIP-IMU sample frames: the reference pose (SIP on all 17 IMUs) next to SOP, SIP and DIP, each given 6 ...
From Figure 9: DIP-IMU sample frames: the reference pose (SIP on all 17 IMUs) next to SOP, SIP and DIP, each given 6 IMUs.
Abstract

We demonstrate a novel deep neural network capable of reconstructing human full body pose in real-time from 6 Inertial Measurement Units (IMUs) worn on the user's body. In doing so, we address several difficult challenges. First, the problem is severely under-constrained as multiple pose parameters produce the same IMU orientations. Second, capturing IMU data in conjunction with ground-truth poses is expensive and difficult to do in many target application scenarios (e.g., outdoors). Third, modeling temporal dependencies through non-linear optimization has proven effective in prior work but makes real-time prediction infeasible. To address this important limitation, we learn the temporal pose priors using deep learning. To learn from sufficient data, we synthesize IMU data from motion capture datasets. A bi-directional RNN architecture leverages past and future information that is available at training time. At test time, we deploy the network in a sliding window fashion, retaining real time capabilities. To evaluate our method, we recorded DIP-IMU, a dataset consisting of $10$ subjects wearing 17 IMUs for validation in $64$ sequences with $330\,000$ time instants; this constitutes the largest IMU dataset publicly available. We quantitatively evaluate our approach on multiple datasets and show results from a real-time implementation. DIP-IMU and the code are available for research purposes.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.