Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning

Ulysse Côté-Allard, Cheikh Latyr Fall, Alexandre Drouin, Alexandre Campeau-Lecours, Clément Gosselin, Kyrre Glette, François Laviolette, Benoit Gosselin

Seminal January 2018

TL;DR Pretrain a ConvNet on Myo-armband EMG from 19 people, then give each new user a second network that learns only the difference. This transfer learning improved all three input types tested, reaching 98.31% on 7 gestures over 17 new participants and 68.98% on 18 NinaPro gestures.

What they do

Deep networks need tens of thousands of examples, but one person can record only a few minutes of surface EMG per sitting. The authors pool recordings from many people instead. A ConvNet is pretrained on 19 people wearing the 8-channel, 200 Hz Myo armband, then adapted to each new user from as little as one 35 s cycle of 7 gestures. They release this 36-participant Myo Dataset and test three inputs: raw EMG, spectrograms and the continuous wavelet transform (CWT).

How it works

Input. 260 ms windows on 8 channels. Armband rotation is aligned across subjects by circularly shifting channels to match each gesture’s most active channel.
Networks. The spectrogram and CWT ConvNets use a slow-fusion video architecture over Time x Channel x Frequency (or Scale). The raw-EMG ConvNet extends an earlier Myo network.
Source Network. Shared by all pretraining subjects, with separate batch-norm statistics per subject. After pretraining, only its batch-norm parameters stay trainable.
Target Network. A new Second Network trains on the new user. Each Source Network layer is scaled by learned coefficients and added to it, so it only learns a residual.

Results

Evaluation set (17 people): transfer learning significantly improved every ConvNet. CWT+TL reached 98.31% vs 97.95% without TL with 4 training cycles, and 94.69% vs 89.02% with 1 cycle. It ranked best against four hand-crafted feature sets on five classifiers.
NinaPro DB5 (18 gestures, 10 people): raw+TL reached 68.98%, vs 66.32% without TL and 62.30% for the best feature set.
Pretraining also significantly helped on 11 NinaPro gestures it had never seen.
14-day real-time study without recalibration: accuracy dropped for the 4 users without feedback, while the 4 who saw live predictions seemingly counteracted the drop.

Limitations

There were no amputee participants, and electrode shift and limb position were not studied. The scheme assumes the new user wears as many electrodes as the pretraining group. Real-time accuracy was below offline (95.42% vs 98.31%), which the authors attribute to reaction delay and to gesture transitions missing from training.

Figures

From Figure 1: The 7 hand/wrist gestures recorded in the Myo Dataset.

From Figure 8: The transfer-learning architecture, shown for the spectrogram ConvNet. Each layer of the pretrained So... — From Figure 8: The transfer-learning architecture, shown for the spectrogram ConvNet. Each layer of the pretrained Source Network (convolution stages C1-C3, fully connected F.C.4-5) is scaled by learned coefficients (S1-S5) and summed element-wise into the matching layer of the Second Network.

From Figure 9: Accuracy of the CWT ConvNet with and without transfer learning on the 11 NinaPro DB5 gestures absent f... — From Figure 9: Accuracy of the CWT ConvNet with and without transfer learning on the 11 NinaPro DB5 gestures absent from pretraining, by number of training repetitions. Error bars are the STD across ten participants.

From Figure 10: Real-time accuracy of the CWT+TL ConvNet over 14 days without recalibration, for the Feedback (blue) ... — From Figure 10: Real-time accuracy of the CWT+TL ConvNet over 14 days without recalibration, for the Feedback (blue) and Without Feedback (orange) groups, with linear fits and 95% bootstrap confidence bands.

Abstract

In recent years, deep learning algorithms have become increasingly more prominent for their unparalleled ability to automatically learn discriminant features from large amounts of data. However, within the field of electromyography-based gesture recognition, deep learning algorithms are seldom employed as they require an unreasonable amount of effort from a single person, to generate tens of thousands of examples. This work's hypothesis is that general, informative features can be learned from the large amounts of data generated by aggregating the signals of multiple users, thus reducing the recording burden while enhancing gesture recognition. Consequently, this paper proposes applying transfer learning on aggregated data from multiple users, while leveraging the capacity of deep learning algorithms to learn discriminant features from large datasets. Two datasets comprised of 19 and 17 able-bodied participants respectively (the first one is employed for pre-training) were recorded for this work, using the Myo Armband. A third Myo Armband dataset was taken from the NinaPro database and is comprised of 10 able-bodied participants. Three different deep learning networks employing three different modalities as input (raw EMG, Spectrograms and Continuous Wavelet Transform (CWT)) are tested on the second and third dataset. The proposed transfer learning scheme is shown to systematically and significantly enhance the performance for all three networks on the two datasets, achieving an offline accuracy of 98.31% for 7 gestures over 17 participants for the CWT-based ConvNet and 68.98% for 18 gestures over 10 participants for the raw EMG-based ConvNet. Finally, a use-case study employing eight able-bodied participants suggests that real-time feedback allows users to adapt their muscle activation strategy which reduces the degradation in accuracy normally experienced over time.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong