© ATRThe drawing illustrates the "visual image reconstruction" technology developed by the Advanced Telecommunications Research Institute International (ATR) and others, which reconstructs figures seen by a subject into images by measuring human brain activity.
The group was led by the Advanced Telecommunications Research Institute International (ATR) and National Institute of Information and Communications Technology (NICT).
By using a functional magnetic resonance imaging (fMRI) system, the new technology measures the patterns of brain activity in the cerebral visual cortex invoked by image information entered through eyes. The field of vision is divided into small areas, and the contrast in each area is estimated based on the corresponding brain activity pattern.
Visual images are reconstructed by combining the estimated contrast values. The error was reduced by combining the estimated values obtained from the assumption that the field of vision is divided in a number of different resolutions.
In the research, 440 images were presented to the subject so that the program can learn the correlation between each image and the brain activity. The image was treated as a combination of small unit elements, and the view angle of about 1° (1.7cm when seen from 1m away) was set as one pixel. The correlation between the image and the brain activity was input to the program for each of the cases where three different unit elements were used, ie, 1 pixel (row) x 2 pixels (column), 2 pixels (row) x 1 pixel (column) and 2 pixels (row) x 2 pixels (column).
Based on the program to which the correlation was input, the image seen by the subject was reconstructed into a 10 x 10 pixel image. Images were accurately constructed even when the subject saw figures or alphabetical letters that were not used in the learning of brain activity patterns, according to ATR.
It was also possible to identify the correct image among more than 100 million candidates, the institute said. Furthermore, the changes of the presented image can be played back as video by utilizing the fMRI signal that is updated every two seconds.
Visual information is converted into electric signals by the retina and then processed by the neurons in the cerebral region called visual cortex, which is located at the back of the head. The visual cortex has a hierarchical structure composed of regions called the primary, secondary visual cortices, etc located in this order from the side closer to the input from the retina.
In the research, the highest reconstruction accuracy was achieved when the brain activity in the primary visual cortex was used. And the higher the order of visual cortex, the lower the accuracy, according to ATR. It was also found that, in the primary visual cortex, a greater amount of information was represented by the patterns between the brain activity signals than by the intensity of individual signals, when compared with visual cortices on other orders.
The technology was developed through a joint-research by ATR, NICT, the Nara Institute of Science and Technology and Japan's National Institute of Natural Science. The achievement will be published in the Dec 11, 2008 edition of a US magazine "Neuron." The authors and the tile of the article are as follows.
Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C. Tanabe, Norihiro Sadato, Yukiyasu Kamitani. "Visual image reconstruction from human brain activity using a combination of multi-scale local image decoders"
Reader Comments
to our Newsletter