A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder

Daehyung Park, Yuuna Hoshi, Charles C. Kemp

Seminal November 2017

TL;DR An LSTM-based variational autoencoder fuses 17 raw sensor signals from a PR2 feeding robot and flags an anomaly when the current observation is poorly reconstructed relative to a state-dependent threshold. Across 12 types of feeding anomalies it reaches an AUC of 0.8710, above five baseline detectors.

What they do

A feeding robot should stop when something unusual happens, such as the user bumping the utensil, the face being occluded or a collision with the environment. The authors train a detector only on successful feedings with a PR2 and flag anomalies online from 17 raw signals (sound energy, utensil force, 7 joint torques, spoon position and mouth position) instead of hand-engineered features.

How it works

LSTM-VAE. At each time step an LSTM encoder maps the input to a 3-D Gaussian latent, and an LSTM decoder outputs a mean and diagonal covariance for the input. Training adds Gaussian noise to the inputs (a denoising criterion).
Progress-based prior. Instead of a fixed standard normal, the prior’s center moves gradually from p1 to pT as the task progresses.
Score and threshold. The anomaly score is the negative log-likelihood of the observation under its reconstructed distribution. A support vector regressor, fit on non-anomalous validation data, maps the latent state to the expected score; the threshold is that score plus a sensitivity constant c.
Data. 1,555 executions from 24 able-bodied participants with 12 anomaly types from fault tree analysis. Leave-one-person-out testing uses 352 executions (160 anomalous) from 8 participants; 1,203 non-anomalous executions with other foods and utensils serve for pre-training.

Results

With 17 raw signals, AUC is 0.8710 versus 0.8075 for EncDec-AD, 0.8012 for an autoencoder, 0.7376 for a one-class SVM and 0.5052 for random. Their earlier HMM-GP detector could not be trained on the 17-D input (underflow errors).
With 4 hand-engineered features, AUC is 0.8564 versus 0.8123 (autoencoder) and 0.8121 (HMM-GP), so raw signals beat the hand-made features.
The state-based threshold gives higher true positive rates than a fixed threshold at the same false positive rates.

Limitations

The paper does not discuss limitations. The test set uses one food (yogurt) and one utensil (silicone spoon), participants were able-bodied and told to keep their upper bodies still, and the anomalies were staged on instruction. The detector outputs only whether the execution is anomalous, not which anomaly occurred.

Figures

From Figure 1: The PR2 feeding system, which collects 17 sensory signals from a wrist camera, microphone, force senso... — From Figure 1: The PR2 feeding system, which collects 17 sensory signals from a wrist camera, microphone, force sensor, joint encoders and current sensors.

From Figure 2: The unrolled LSTM-VAE detector. It is trained on multimodal signals with progress-based priors, and a ... — From Figure 2: The unrolled LSTM-VAE detector. It is trained on multimodal signals with progress-based priors, and a threshold estimator is trained on its outputs; at test time it reports an anomaly when the anomaly score exceeds the estimated threshold.

From Figure 5: The 12 anomaly types used in the experiments, caused by the user, the environment or the system.

From Figure 8: ROC curves of the LSTM-VAE detector with a state-based threshold and with a fixed threshold, against a... — From Figure 8: ROC curves of the LSTM-VAE detector with a state-based threshold and with a fixed threshold, against a random classifier.

Abstract

The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem. We introduce a long short-term memory based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution. We also introduce an LSTM-VAE-based detector using a reconstruction-based anomaly score and a state-based threshold. For evaluations with 1,555 robot-assisted feeding executions including 12 representative types of anomalies, our detector had a higher area under the receiver operating characteristic curve (AUC) of 0.8710 than 5 other baseline detectors from the literature. We also show the multimodal fusion through the LSTM-VAE is effective by comparing our detector with 17 raw sensory signals versus 4 hand-engineered features.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong