Reconstructing Visuals from Brain Signals Using Generative AI

Cognitive neuroscientists are now capable of reconstructing the exact images a person is looking at by translating their brain activity using generative artificial intelligence. This article provides a comprehensive overview of how researchers capture functional magnetic resonance imaging (fMRI) blood-oxygen-level-dependent (BOLD) signals, map this neural data to the latent spaces of pre-trained diffusion models, and combine structural and semantic brain information to generate high-resolution visual reconstructions of human perception.

Capturing Visual Data via fMRI BOLD Signals

When a human views an image, neurons in the visual cortex fire rapidly, consuming energy and requiring an influx of oxygenated blood. Functional magnetic resonance imaging (fMRI) detects these changes using blood-oxygen-level-dependent (BOLD) contrast.

As a subject looks at various photographs inside an MRI scanner, the machine records the BOLD signals across thousands of tiny 3D brain units called voxels. Different areas of the visual cortex process different features; the early visual cortex (V1 to V4) processes low-level structural details like lines, edges, and colors, while the higher visual cortex processes high-level semantic concepts like faces, animals, and places. This recorded voxel data serves as the raw input for the AI models.

Mapping Brain Activity to AI Latent Spaces

Directly translating noisy, low-resolution fMRI data into a crisp image is nearly impossible without an intermediary translator. To solve this, neuroscientists use "latent spaces"—the compressed mathematical representations of data used by generative AI models like Stable Diffusion.

Instead of training an AI from scratch, researchers train simple linear regression models (decoders) to translate the fMRI BOLD signals into the exact mathematical language the AI already understands. The subject views thousands of training images while in the scanner. The researchers then map the subject's specific brain patterns to the corresponding image embeddings within the AI's latent space. Once trained, this decoder can take a brand-new fMRI scan and predict the exact latent representations the AI needs to generate the unseen image.

Combining Structural and Semantic Pathways

To achieve accurate reconstructions, neuroscientists split the fMRI data into two distinct pathways that mirror how both the brain and diffusion models work:

• The Structural Pathway: Researchers take BOLD signals from the early visual cortex and map them to the variational autoencoder (VAE) of the diffusion model. This produces a blurry, low-level rough draft of the image that accurately captures the layout, shapes, colors, and orientation of the original picture.
• The Semantic Pathway: Researchers take BOLD signals from the higher visual cortex and map them to a Contrastive Language-Image Pre-training (CLIP) model. This translates the brain activity into high-level semantic descriptions (e.g., "a clock tower against a blue sky") without needing to know where objects are physically located in the frame.

The Diffusion Reconstruction Process

The final step relies on the core mechanism of latent diffusion models, which generate images by iteratively removing noise from a randomized static picture.

To reconstruct the perceived image, the diffusion model starts with the blurry structural rough draft decoded from the early visual cortex instead of pure random noise. This ensures the final image retains the correct spatial layout. Next, the model uses the CLIP semantic data decoded from the higher visual cortex to guide the "denoising" process.

As the diffusion model sharpens the image step by step, the semantic guidance tells the AI how to fill in the blurry shapes—turning a vague vertical rectangle into a detailed clock tower. Within seconds, the generative model outputs a photorealistic image that closely matches both the visual layout and the conceptual meaning of the picture the human subject originally saw.