This AI finds patterns in visual brain activity and turns them into pictures

Scientists have developed an artificial intelligence (AI) tool capable of recreating images based on brain scans with remarkable accuracy. The system, named Brain-IT, utilizes data from functional magnetic resonance imaging (fMRI) to measure activity in the visual cortex, the area of the brain responsible for processing visual input.

Researchers created a computer model that predicts both the overall layout of an image and high-level details about its content. This predicted information is then fed into an AI image generator.

Brain-IT builds upon a concept called the Universal Brain Encoder, which begins with an image and subsequently predicts the fMRI response expected in individual brain voxels, or small 3D units of tissue. Each voxel possesses a unique vector, capturing its specific response to various visual elements within images, such as edges, textures, locations, or abstract concepts like faces and scenes.

Traditional image-to-fMRI systems typically require custom training for each participant and dataset. In contrast, the Universal Brain Encoder is more adaptable, learning from data from multiple individuals and transferring this knowledge across participants, datasets, or MRI systems with varying magnetic-field strengths.

The AI model begins with measured brain activity and reconstructs the image the individual viewed. It categorizes thousands of tiny brain-scan measurements from the visual cortex into 128 groups based on the types of visual information they typically carry. These groups then exchange information, enabling the system to combine insights about a scene's shape, color, and possible objects to construct a more comprehensive image representation.

The decoder generates two types of visual predictions. The first captures semantic information, which includes features associated with what is likely present in a scene, such as the characteristics of a dog and an outdoor, grassy setting in an image of a dog standing in grass. The second type captures structural information, like the broad visual layout or arrangement of forms. In the example provided, this aspect would involve the outline of the dog and its position in the frame.

The structural prediction aids the image generator, providing a starting point for the image, rather than beginning from a random background. Meanwhile, semantic information guides the generator in refining the image by removing any remaining static until the final image emerges. However, it is crucial to note that while Brain-IT can recreate what people see, it cannot reproduce memories or imagined scenes.

The images generated from fMRI scans rely on participants' cooperation and full collaboration, and the original image viewed must be shown to the model for the system to function correctly.