Decoding speech perception from non-invasive brain recordings

Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, Jean-Rémi King

Seminal August 2022

TL;DR A convolutional network trained with a contrastive loss matches 3 s of MEG or EEG to wav2vec 2.0 features of the speech a person is hearing. Across 175 volunteers it picks the right segment out of more than 1,000 with up to 41% top-1 accuracy for MEG.

What they do

Speech decoders that work well so far rely on electrodes implanted in the brain. This paper decodes perceived speech from non-invasive recordings instead. Given 3 s of MEG or EEG recorded while a healthy volunteer listens to stories or sentences, the model identifies which speech segment was heard among all segments of the test set. It is trained and evaluated on four public datasets, two MEG and two EEG, covering 175 participants.

How it works

Speech module. A wav2vec 2.0 model pretrained on 56k hours of speech turns each 3 s audio segment into a latent representation (the average of its last four transformer layers).
Brain module. Raw M/EEG, resampled to 120 Hz, goes through a spatial attention layer over sensor positions, a subject-specific 1x1 convolution (the “Subject Layer”) and five blocks of dilated residual convolutions. One model is trained across all participants of a dataset.
Contrastive loss. Instead of regressing the Mel spectrogram, a CLIP loss aligns brain and speech latents, with the other segments in the batch as negatives. At test time the candidates are all test segments, so segments never seen in training can be decoded.
Minimal preprocessing. Baseline correction, a robust scaler and clamping at 20 standard deviations.

Results

MEG: 70.7% (Gwilliams) and 67.5% (Schoffelen) top-10 segment accuracy out of more than 1,000 candidates, and up to 41.3% top-1; the best participant exceeds 80% top-1. A random model scores below 1% top-10.
EEG is much lower: 25.7% (Brennan) and 17.7% (Broderick) top-10. The gap remains after matching sensors, subjects and recording time.
On Gwilliams, top-10 rises from 12.4% (Mel regression) to 55.1% with the CLIP loss and 70.7% with wav2vec 2.0 targets.
Removing the Subject Layer drops MEG top-10 to 42.4-47.0%; removing clamping collapses training. Accuracy grows with the number of training participants.

Limitations

The model decodes speech perception in healthy listeners, not speech production in patients. The contrastive objective can only pick from a predetermined set of candidates, so a generative module is still needed. EEG performance is modest, zero-shot word decoding falls sharply for EEG, and MEG systems are not commonly deployed in clinics.

Figures

From Figure 1: Method. A pretrained speech module (wav2vec 2.0) extracts representations of 3 s speech segments, and ... — From Figure 1: Method. A pretrained speech module (wav2vec 2.0) extracts representations of 3 s speech segments, and a deep convolutional brain module learns representations of the matching 3 s MEG or EEG window that align with them under a contrastive (CLIP) loss. At evaluation, the model computes the probability of each speech segment given the brain recording, so decoded segments need not be present in the training set.

From Figure 2: Design choices. A. A 3 s speech segment and its Mel spectrogram. B. Mel spectrogram predicted with a d... — From Figure 2: Design choices. A. A 3 s speech segment and its Mel spectrogram. B. Mel spectrogram predicted with a direct regression loss. C. Replacing the regression loss with a CLIP loss improves reconstruction. D. Replacing the Mel spectrogram target with wav2vec 2.0. E. Architecture of the brain module: a spatial attention layer, a 1x1 convolution, a subject-specific layer, five blocks of dilated residual convolutions and two final 1x1 convolutions.

From Figure 3: A. Each dot is the top-10 accuracy of a single subject, estimated with the full test set (blue) or wit... — From Figure 3: A. Each dot is the top-10 accuracy of a single subject, estimated with the full test set (blue) or with 50 possible segments (orange). B. Same as A, for top-1 accuracy. C. Top-10 accuracy as a function of the number of participants in the training set, evaluated on the first 10% of the participants.

From Figure 5: The R values quantify how well phonemes, word frequency, part-of-speech, word embedding and phrase emb... — From Figure 5: The R values quantify how well phonemes, word frequency, part-of-speech, word embedding and phrase embedding each predict the probability of the predicted word being correct. Error bars are the SEM across participants.

Abstract

Decoding speech from brain activity is a long-awaited goal in both healthcare and neuroscience. Invasive devices have recently led to major milestones in that regard: deep learning algorithms trained on intracranial recordings now start to decode elementary linguistic features (e.g. letters, words, spectrograms). However, extending this approach to natural speech and non-invasive brain recordings remains a major challenge. Here, we introduce a model trained with contrastive-learning to decode self-supervised representations of perceived speech from the non-invasive recordings of a large cohort of healthy individuals. To evaluate this approach, we curate and integrate four public datasets, encompassing 175 volunteers recorded with magneto- or electro-encephalography (M/EEG), while they listened to short stories and isolated sentences. The results show that our model can identify, from 3 seconds of MEG signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities on average across participants, and more than 80% in the very best participants - a performance that allows the decoding of words and phrases absent from the training set. The comparison of our model to a variety of baselines highlights the importance of (i) a contrastive objective, (ii) pretrained representations of speech and (iii) a common convolutional architecture simultaneously trained across multiple participants. Finally, the analysis of the decoder's predictions suggests that they primarily depend on lexical and contextual semantic representations. Overall, this effective decoding of perceived speech from non-invasive recordings delineates a promising path to decode language from brain activity, without putting patients at risk for brain surgery.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong