Deep learning with convolutional neural networks for EEG decoding and visualization

Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, Tonio Ball

Seminal March 2017

TL;DR Convolutional networks trained end-to-end on raw EEG decode imagined and executed movements as well as filter bank common spatial patterns (FBCSP), the standard hand-engineered method, with mean accuracy of 84.0% against 82.1%. New visualizations show the networks learn to use alpha, beta and high-gamma band power.

What they do

EEG movement decoding has relied on hand-designed features, above all filter bank common spatial patterns (FBCSP), which extracts band power. This study asks whether ConvNets trained end-to-end on raw EEG can match it, which design and training choices matter, and what the networks learn. It compares a shallow, a deep, a hybrid and a residual ConvNet with a validated FBCSP baseline on two four-class datasets of imagined and executed movements.

How it works

Input and first block. A trial is a 2D array of electrodes by time steps. Every network starts with a temporal convolution followed by a spatial convolution over all electrodes, analogous to FBCSP’s bandpass and spatial filters.
Architectures. The deep ConvNet stacks four convolution-max-pooling blocks with ELUs, batch normalization and dropout. The shallow ConvNet adds squaring, mean pooling and a logarithm, which mirror FBCSP’s log-variance features.
Cropped training. Rather than whole 4.5 s trials, the networks train on about 2 s sliding windows, 625 crops per trial, computed in one forward pass; the crop predictions are averaged at test time.
Visualization. Band-power envelopes are correlated with unit outputs, and input amplitudes are perturbed to map how each frequency band and electrode causally changes the predictions.

Results

Mean accuracy over both datasets (0 Hz upward): 82.1% for FBCSP, 84.0% for the deep ConvNet (+1.9%, p<0.05) and +1.1% for the shallow one. On data high-passed at 4 Hz the shallow ConvNet is 3.4% better than FBCSP.
Without batch normalization, dropout and ELUs together, the deep ConvNet is significantly worse than FBCSP (p<0.001). The 31-layer residual network is 3.4% below the deep ConvNet.
Cropped training significantly helps the deep ConvNet on the 4 Hz high-passed data (p<1e-5), and makes no significant difference elsewhere.
Perturbation maps show the deep ConvNet uses alpha, beta and high-gamma power with scalp distributions expected for hand and foot movements.

Limitations

ConvNets did not clearly outperform FBCSP: significant gains never exceeded 3.5%, with large variation between subjects. Training takes 5-45 times longer than FBCSP. Decoding is offline and per subject, the comparison leaves out regularized FBCSP variants, and the visualizations only cover known band power features.

Figures

From Figure 1: Deep ConvNet architecture. EEG input (at the top) is progressively transformed towards the bottom, unt... — From Figure 1: Deep ConvNet architecture. EEG input (at the top) is progressively transformed towards the bottom, until the final classifier output. Black cuboids: inputs/feature maps; brown cuboids: convolution/pooling kernels.

From Figure 2: Shallow ConvNet architecture. Black cuboids: inputs/feature maps; brown cuboids: convolution/pooling k... — From Figure 2: Shallow ConvNet architecture. Black cuboids: inputs/feature maps; brown cuboids: convolution/pooling kernels.

From Figure 6: FBCSP vs. ConvNet decoding accuracies. Each small marker represents accuracy of one subject, the large... — From Figure 6: FBCSP vs. ConvNet decoding accuracies. Each small marker represents accuracy of one subject, the large square markers represent average accuracies across all subjects of both datasets. Markers above the dashed line indicate experiments where ConvNets performed better than FBCSP.

From Figure 16: Input-perturbation network-prediction correlation maps for the deep ConvNet. Correlation of class pre... — From Figure 16: Input-perturbation network-prediction correlation maps for the deep ConvNet. Correlation of class predictions and amplitude changes, averaged over all subjects of the High-Gamma Dataset.

Abstract

Deep learning with convolutional neural networks (deep ConvNets) has revolutionized computer vision through end-to-end learning, i.e. learning from the raw data. Now, there is increasing interest in using deep ConvNets for end-to-end EEG analysis. However, little is known about many important aspects of how to design and train ConvNets for end-to-end EEG decoding, and there is still a lack of techniques to visualize the informative EEG features the ConvNets learn. Here, we studied deep ConvNets with a range of different architectures, designed for decoding imagined or executed movements from raw EEG. Our results show that recent advances from the machine learning field, including batch normalization and exponential linear units, together with a cropped training strategy, boosted the deep ConvNets decoding performance, reaching or surpassing that of the widely-used filter bank common spatial patterns (FBCSP) decoding algorithm. While FBCSP is designed to use spectral power modulations, the features used by ConvNets are not fixed a priori. Our novel methods for visualizing the learned features demonstrated that ConvNets indeed learned to use spectral power modulations in the alpha, beta and high gamma frequencies. These methods also proved useful as a technique for spatially mapping the learned features, revealing the topography of the causal contributions of features in different frequency bands to decoding the movement classes. Our study thus shows how to design and train ConvNets to decode movement-related information from the raw EEG without handcrafted features and highlights the potential of deep ConvNets combined with advanced visualization techniques for EEG-based brain mapping.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong