Viewing an action — for example, biting or kicking or punching — in a photo versus a video doesn’t change the understanding of what’s taking place, according to new research from University of Pennsylvania psychologists Russell Epstein, John Trueswell and Alon Hafri, who published their work in The Journal of Neuroscience.

“We were able to elicit similar representations in people’s brains using very different visual stimuli, movies and still images,” said Epstein, a professor of psychology in the School of Arts and Sciences. “We think that these might be the building blocks for people’s concepts of such actions.” 

The research stemmed from previous work done in Epstein’s lab focused on object and scene recognition. Hafri, a fourth-year graduate student, led this newest project, which goes a step further.

​​​​​​​“An action looks different every time you see it,” Hafri said. So “we started out by asking, where in the brain can we find consistent encoding of information about actions?”

Collaborating with Trueswell, a psychology professor who focuses on language and thought, the team devised an experiment that not only looked at which brain areas came into play when observing actions but also whether the neural codes changed depending on visual input. They created two sets of stimuli.

The first, of dynamic videos, showcased the same two actors from the same vantage point in front of the same indoor setting. The only aspect that changed was the action taking place. The second, of static images, offered great variety in terms of viewpoint, context and setting. Unlike the videos, they contained no motion.

Hafri and colleagues then recruited 15 college-age adults to participate in an fMRI study. The participants viewed the stimuli in an MRI machine while performing an unrelated task to keep them engaged. And the researchers focused on brain activity when participants viewed each stimulus.

“We have data about what their brain activity patterns looked like while they were viewing every image and every video,” Hafri said.

By comparing activity patterns when participants viewed a given movement as an image and video, the researchers could ask where in the brain the same patterns arose across the two stimulus sets. Were they the same, for example, when a participant saw a video of a person eating a sandwich compared to an image of a dog chewing a bone? The answer, it turns out, was yes.

“Brain representations we observed,” Hafri said, “are surprisingly consistent no matter how different biting looks in the world. Even when you don’t see the whole action take place, like when you view a static image, these brain regions are encoding similarly to when you see the whole action from beginning to end, as in a video. That suggests they’re doing so at a very high level.”

​​​​​​​​​​​​​​​​​​​​​The researchers also confirmed what previous work had shown: That parts of the brain within the inferior parietal, occipitotemporal and premotor cortex, responsible, broadly speaking, for aspects of spatial cognition, high-level vision and planning movements, respectively, play an important role in observing actions.

“We’re not the first to identify this network of brain regions, but our results provide new evidence that these areas are representing actions in an abstract manner,” Epstein said.

Finally, the results verify the idea that, even if our vision gets occluded, cutting off part of an action sequence, the brain can complete the picture.

“It would be pretty detrimental if, any time we closed our eyes for a moment or someone walked in front of us, we were no longer able to infer what was happening,” Hafri said. “The fact that we have neural systems that code for actions despite missing visual input is pretty important.”

Future research will likely continue along these lines, studying whether reading about an action results in the same brain activity as seeing it does.

Funding for this research came from the Center for Functional Neuroimaging at the University of Pennsylvania, National Science Foundation and National Institutes of Health.