Brain decoding: toward real-time reconstruction of visual perception

Yohann Benchetrit, Hubert Banville, Jean-Rémi King

Seminal October 2023

Images shown to participants for one second next to the outputs decoded from their MEG, shown at 1/4 speed.

TL;DR Decodes the images a person is seeing from MEG recordings by aligning brain activity to pretrained image embeddings and feeding the predictions to a diffusion model. The deep MEG decoder retrieves images 7X better than linear decoders, reaching about 70% top-5 accuracy on a 200-image test set.

What they do

Image reconstruction from brain activity has mostly used fMRI, which yields about one snapshot every two seconds. This paper decodes seen images from magnetoencephalography (MEG), which is much faster but has only about 300 sensors. A brain module is trained to map MEG windows onto pretrained image embeddings, which are then used to retrieve the seen image or to condition a diffusion model that generates it.

How it works

Brain module. A dilated residual ConvNet with a spatial attention layer and a per-subject linear layer maps a 272-channel MEG window to one image latent. A learned affine projection aggregates over time, followed by two MLP heads.
Objectives. A CLIP (contrastive) loss for retrieval and an MSE loss for regressing latents, combined with a tuned weight.
Generation. Three brain modules predict CLIP-Vision, CLIP-Text and AutoKL latents, which condition a pretrained latent diffusion model.
Data. THINGS-MEG: four participants and 22,448 unique images. Categories of the test images are removed from training, so test categories are unseen.

Results

Retrieval improves 7X over linear Ridge baselines. Top-5 accuracy on the 200-image test set is 70.33% with VGG-19, 68.66% with CLIP-Vision and 68.00% with DINOv2 embeddings (chance 2.5%); about 13% on the 2,400-image large test set (chance 0.21%).
With 100 ms sliding windows, retrieval peaks for windows ending 200-275 ms after image onset and again 150-200 ms after image offset; DINOv2 is particularly strong after offset.
Generated images often preserve the category of the seen image but few low-level features: PixCorr is 0.076 for MEG against 0.305 for a similar pipeline on 7T fMRI, and the CLIP 2-way score is 0.767 against 0.917.

Limitations

MEG generations preserve low-level features such as object position and color worse than 7T fMRI. The method relies on several pretrained models and only learns the MEG alignment end-to-end; fine-tuning the image and generation modules is untested. Reported generation metrics average predictions over repeated presentations of each image, and clinical use would likely require extending the work to EEG.

Videos

Viewed images next to the images predicted from MEG.

Figures

From Figure 1: (A) Approach. Locks indicate pretrained models. (B) Processing schemes. Unlike image generation, retri... — From Figure 1: (A) Approach. Locks indicate pretrained models. (B) Processing schemes. Unlike image generation, retrieval happens in latent space, but requires the true image in the retrieval set.

From Figure 2: Image retrieval performance obtained from a trained deep ConvNet. Linear decoder baseline performance ... — From Figure 2: Image retrieval performance obtained from a trained deep ConvNet. Linear decoder baseline performance is shown with a black transparent bar for each latent. The small test set comprises 200 distinct images, each belonging to a different category; the large test set comprises 12 images from each of those 200 categories. The best latent representations yield accuracies around 70% and 13% for the small and large test sets, respectively.

From Figure 4: Handpicked examples of successful generations. (A) Generations obtained on growing windows starting at... — From Figure 4: Handpicked examples of successful generations. (A) Generations obtained on growing windows starting at image onset (0 ms) and ending at the specified time. (B) Full-window generations (-500 to 1,000 ms).

Abstract

In the past five years, the use of generative and foundational AI systems has greatly improved the decoding of brain activity. Visual perception, in particular, can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from a limited temporal resolution ($\approx$0.5 Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalography (MEG), a neuroimaging device capable of measuring brain activity with high temporal resolution ($\approx$5,000 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) a pretrained image generator. Our results are threefold: Firstly, our MEG decoder shows a 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses to images are best decoded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that high-level visual features can be decoded from MEG signals, although the same approach applied to 7T fMRI also recovers better low-level features. Overall, these results, while preliminary, provide an important step towards the decoding -- in real-time -- of the visual processes continuously unfolding within the human brain.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong