Brain-to-Text Decoding: A Non-invasive Approach via Typing

Jarod Lévy, Mingfang Zhang, Svetlana Pinet, Jérémy Rapin, Hubert Banville, Stéphane d'Ascoli, Jean-Rémi King

Seminal February 2025

TL;DR Brain2Qwerty decodes the sentences people type from non-invasive brain recordings, using a convolutional module, a transformer and a character-level language model. With MEG it reaches a 32% character error rate on average (19% for the best participant), against 67% with EEG.

What they do

Brain-to-text neuroprostheses work well but need brain surgery. This paper asks how far non-invasive recordings can go. 35 healthy, skilled typists read a short Spanish sentence, waited, then typed it from memory on a QWERTY keyboard while EEG or MEG was recorded. Brain2Qwerty, a three-stage deep network, decodes the typed characters from those signals.

How it works

Data. Each session has 128 unique sentences of 5 to 8 words, typed with no backspace and no on-screen text. This gives 146K characters of EEG and 193K of MEG (20 participants each). MEG needed a custom keyboard with non-ferromagnetic springs.
Convolutional module. A 0.5 s window of signal around each key press goes through spatial attention over sensors, a subject-specific linear layer and 8 convolutional blocks, producing one embedding per keystroke.
Transformer. Four layers refine the keystroke predictions using the rest of the sentence and output logits over 29 character classes.
Language model. A 9-gram character model pretrained on Spanish Wikipedia is combined with the logits in a beam search.

Results

Character error rate (CER) is 32% with MEG and 67% with EEG. The best and worst MEG participants reach 19% and 45%.
Against EEGNet, Brain2Qwerty improves CER 2.25-fold on MEG and 1.14-fold on EEG. The transformer and the language model each lower CER further.
Some held-out sentences are decoded perfectly, including ones the participant mistyped. Words absent from training are decoded poorly (68% CER).
Decoding errors follow the keyboard: confusion rate correlates with key distance (R=0.73). CER keeps falling as training data grows.

Limitations

The model is not real-time: it needs the full sentence and windows aligned to keystrokes. It was trained with supervision on healthy people who physically typed, so it does not apply to locked-in patients. Current MEG systems are not wearable, and a gap to invasive interfaces remains.

Figures

From Figure 1: Approach. Sentences were displayed word-by-word on a screen. Following the final word, a visual cue pr... — From Figure 1: Approach. Sentences were displayed word-by-word on a screen. Following the final word, a visual cue prompted participants to begin typing the sentence, without visual feedback. Brain2Qwerty decodes text from brain activity in three stages: a convolutional module, input with 500 ms windows of M/EEG signals, a transformer module trained at the sentence level, and a pretrained language model to correct the outputs of the transformer.

From Figure 2: Decoding performance across models. A, B. Difference in EEG and MEG evoked responses between left and ... — From Figure 2: Decoding performance across models. A, B. Difference in EEG and MEG evoked responses between left and right hand key presses. C, D. Linear classifiers trained at each time sample to predict the hand and the character of each key press. E-H. Comparison of baselines (linear and EEGNet) and ablations of the three-step Brain2Qwerty model, for both hand-error-rate (HER) and character-error-rate (CER). Each point is the average score of a single participant.

From Figure 4: Character- and word-level performance on MEG data with the Conv+Trans model. A. Character-error-rate (... — From Figure 4: Character- and word-level performance on MEG data with the Conv+Trans model. A. Character-error-rate (CER) across part-of-speech categories. B. CER as a function of word frequency, including out-of-vocabulary (OOV) words absent from the training set. C. CER as a function of character frequency. D. CER as a function of recording time included in the training set.

Abstract

Modern neuroprostheses can now restore communication in patients who have lost the ability to speak or move. However, these invasive devices entail risks inherent to neurosurgery. Here, we introduce a non-invasive method to decode the production of sentences from brain activity and demonstrate its efficacy in a cohort of 35 healthy volunteers. For this, we present Brain2Qwerty, a new deep learning architecture trained to decode sentences from either electro- (EEG) or magneto-encephalography (MEG), while participants typed briefly memorized sentences on a QWERTY keyboard. With MEG, Brain2Qwerty reaches, on average, a character-error-rate (CER) of 32% and substantially outperforms EEG (CER: 67%). For the best participants, the model achieves a CER of 19%, and can perfectly decode a variety of sentences outside of the training set. While error analyses suggest that decoding depends on motor processes, the analysis of typographical errors suggests that it also involves higher-level cognitive factors. Overall, these results narrow the gap between invasive and non-invasive methods and thus open the path for developing safe brain-computer interfaces for non-communicating patients.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong