FETTUCCINE: Fast and efficient brain-to-text decoding on mobile devices

TL;DR FETTUCCINE swaps the audio front end of off-the-shelf speech-recognition models (Whisper, Parakeet) for a neural adapter and decodes intracortical speech-motor activity straight to text, with no n-gram or LLM stack. It reaches 4.39% word error rate on the Brain-to-Text 2025 benchmark and runs on an iPhone 15 Pro Max at up to 370x real time.
What they do
Speech neuroprostheses decode attempted speech from electrodes implanted in motor cortex. The standard pipeline (a GRU phoneme decoder, then an n-gram language model and LLM rescoring) needs a workstation with hundreds of GB of RAM, and its latency per sentence ranges from 52 ms to over 10 s. FETTUCCINE instead adapts pretrained automatic speech recognition (ASR) models to take neural features directly and output text end to end. The models are small enough to run on a phone.
How it works
- Neural Adapter. Replaces the ASR model’s audio convolution block with a temporal-convolution U-Net and convolutional subsampling. It maps threshold crossings and spiking-band power from four 64-channel arrays into the ASR embedding space.
- Backbones and training. Whisper (encoder-decoder transformer) and Parakeet (Conformer) variants. The adapter is warmed up alone, then trained jointly with LoRA on the encoder layers, using each model’s native loss; the backbone stays frozen.
- Word-level augmentation. A “neural envelope” estimates word boundaries, which are used to warp, rescale and crop individual words.
- On device. Models are converted to Core ML and run in Swift on an iPhone 15 Pro Max.
Results
- Brain-to-Text 2025 (participant T15): Whisper Small reaches 4.39% / 5.05% WER on the public / private test split. The GRU + 5-gram + OPT-6.7B baseline gets 6.67% / 7.00% and needs 320 GB RAM. The best prior end-to-end results are 11.06% (BIT) and 8.70% (BrainWhisperer).
- On the iPhone every variant stays under 10% validation WER. Whisper Tiny takes 48 ms per sentence (373x real time) at 8.4% WER with 209 MB peak memory; Whisper Base gets 7.0% at 93 ms.
- On an A40 GPU, Whisper Base latency is 50.2 ± 16.2 ms vs 494.8 ± 2457.7 ms for the GRU baseline.
- Models fail zero-shot on later held-out sessions but reach reasonable WER after finetuning on about 15 minutes of calibration data on average.
Limitations
Offline evaluation on recorded data; closed-loop use is untested. WER trails the top benchmark entries (2% or below) and, for a second participant (T12), the cascaded methods (12.98% vs 8.08-9.76%). All variants are non-causal, so text arrives only after the whole sentence. Usable decoding needs roughly 5-10 earlier sessions of data.
Figures



Abstract
Recent advances in speech neuroprostheses have enabled new communication avenues for people who have lost the ability to speak due to neurological illness or injury. These systems can decode neural activity during attempted speech into text by relying on language priors to achieve high decoding accuracy. This performance currently comes at the cost of using a compute-intensive language modeling stack, presenting a major bottleneck for real-world use. Further, current language models incur highly variable and disruptive inference latencies, limiting communication throughput and preventing users from reliably participating in natural conversation. We introduce FETTUCCINE, an end-to-end brain-to-text framework that leverages the speed and portability of automatic speech recognition (ASR) models to address the aforementioned practical challenges, while achieving usable (<10%) word error rates (WER), efficient adaptation to neural non-stationarities, and reliable performance on mobile devices. When evaluated using publicly available data from an intracortical speech neuroprosthesis user, FETTUCCINE outperforms previous end-to-end methods, achieving as low as 4.39% WER. Most notably, our models successfully run on a commercially-available mobile device with throughput reaching up to 370x real-time, providing the first demonstration of brain-to-text decoding on a mobile device. We also show that our models can be successfully finetuned to extend usable performance to future days. Beyond performance, our approach enables localization of the neural features most relevant for decoding via gradient-based saliency maps, which we show align with well-established physiological priors across different ASR model families. Taken together, our results show that FETTUCCINE overcomes core infrastructural and computational barriers, yielding a new class of portable brain-to-text communication.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
