PHILADELPHIA — One of the aims of cognitive neuroscience is to understand how the brain works on two distinct levels: the psychological experience of mental tasks and the underlying neurobiology that enables them. Tools like functional magnetic resonance imaging, or fMRI, have allowed cognitive neuroscientists to connect those two levels for a variety of everyday experiences, such as solving a math problem or remembering a word. Now, researchers at the University of Pennsylvania have used fMRI to make an important finding about how people recognize individual objects and how they can mentally combine them into complex scenes.

The research was conducted by Russell Epstein, an associate professor of psychology at Penn’s Center for Cognitive Neuroscience in the School of Arts and Sciences, and Sean MacEvoy, a former postdoctoral fellow at Penn, now an assistant professor at Boston College.

Their work was published in the journal Nature Neuroscience.

Epstein has long studied the neural basis of scene recognition. More than a decade ago, working with Nancy Kanwisher of the Massachusetts Institute of Technology, he discovered a part of the brain now known as the Parahippocampal Place Area. As its name suggests, the PPA responds strongly when people view images of places — complex scenes such as an intersection or a crowded beach — but only weakly when they view discrete objects, like a stop sign or a beach ball. That response allows people to rapidly recognize what scene they are seeing.   

“I could flash you a picture of an office or a kitchen very briefly, say a few milliseconds, and you'd immediately know which one it is,” Epstein said. “We've known for about 40 years that people can do this, but how people can do this remains a big question within the field of visual cognition.”  

During the last few years, several hypotheses have come out regarding the mechanisms at work in the PPA. In this study, Epstein and MacEvoy tested a hypothesis about a different area of the brain that also processes visual information: the lateral occipital cortex, or LO.

As opposed to the PPA, the LO is active when people look at individual objects, rather than scenes. But thanks to an elegant experiment design, Epstein and MacEvoy have shown that the LO can combine information about key objects into a constructive recognition of a scene.      

“This suggests,” Epstein said, “that there are two different routes to scene recognition, one by the LO on the basis of the objects within the scene, the other by the PPA through the scene’s overall qualities, like the three-dimensional geometry of the space.”

Epstein and MacEvoy tested the LO’s role by choosing four scenes and picking two signature objects that are commonly associated with each of them. Toilets and bathtubs went with bathrooms, refrigerators and ovens with kitchens, cars and traffic lights with intersections and slides and swing-sets with playgrounds.  The researchers selected multiple pictures of each of the scenes and objects, and, though all of the scenes contained examples of their signature objects, none of the object pictures was taking directly from any of the scenes.  

The researchers showed these pictures to experiment participants while an MRI machine scanned their brains. By measuring what parts of the brain showed the most metabolism while looking at each picture, the researchers could see the areas in both the LO and PPA that were working hardest. They could also go a step further by analyzing the patterns of activation in those regions, looking for similarities and differences in the responses to different pictures. Pattern similarities in an individual looking at two different pictures would suggest a common mental representation for both.

While the researches did multiple experiments and comparisons, the most important finding was that the LO patterns produced when an individual looked at a scene resembled the average of the patterns produced when that person looked at that scene’s two signature objects.

“If I take the patterns that an individual has when looking at stoves and average it together with the patterns that an individual has when looking at refrigerators and then compare it to the pattern that comes from looking at kitchens, it's more similar to the kitchen pattern that is the patterns of any of the other scenes,” Epstein said.

The PPA scene patterns showed no similarity to the patterns of their related objects.  

“The patterns for both areas distinguish between the types of scenes, but it's only in the LO where the scene representations are built out of the object representations,” Epstein said. “We still think the main purpose of the LO is to recognize objects, but it can also recognize scenes by making an average representation of all the objects there. The LO can do double duty, depending on whether you are paying attention to one object or attending generally to the entire scene.”

Though more research is needed to determine exactly how and why the brain splits up scene recognition between the LO and PPA, there are many situations where having two parallel systems could potentially come in handy. 

“It may be that PPA scene recognition is good enough most of the time, but there are situations where you need information about the objects to distinguish between two similar scenes,” Epstein said. “The difference between a living room and a bedroom, for example, is all in the objects.”

This work was supported by National Eye Institute of the National Institutes of Health.