A study published in Economics Letters co-authored by Alex Rees-Jones, an associate professor of business economics and public policy at the Wharton School, and led by Douglas M. McKee and George Orlov of Cornell University, found that online learning during the COVID-19 pandemic hurt student learning, but did not hurt particular demographic groups more than others. However, they did find that if the instructor used active learning techniques, students were more engaged and thus learning outcomes improved.
In this project, the researchers were studying the impact of the switch to online teaching on student performance during the beginning of the pandemic in spring 2020.
“I was one of a group of professors who were part of a multi-year program meant to assess and improve active learning techniques in the classroom,” says Rees-Jones. “We were running standardized tests at the end of each semester so we could see the effect of changes. By chance, COVID happened during all of this, so the cross-semester system we built to measure changes in student learning could be used to assess what happened from COVID.”
The bottom line to the study, according to Rees-Jones, is that the pandemic hurt student learning quite a bit.
“We studied if it hurt particular demographic groups more than others, but found no evidence that this mattered in our context,” he says. “One thing that did matter, though, was the instructor’s use of active learning techniques. Using approaches built to improve student engagement mitigated a lot of the negative effects, and not using any of those techniques was associated with quite bad outcomes.”
The researchers compared student performance on standard assessments in spring 2020 to student performance in the same courses in either fall or spring 2019 to estimate the impact of the emergency switch to remote instruction induced by the COVID-19 pandemic. Using these data, they addressed three questions:
First, they examined how end-of-semester knowledge was influenced by the measures taken in spring 2020.
The typical difficulty in assessing a question like this is finding measures of learning that are comparable over time, according to Rees-Jones.
“For example, if the average grade on the final exam in one semester is an A and the average another semester is a B, you don’t actually know if the amount learned across semesters was different,” he says. “One final could simply have harder questions.”
To get around this issue, professors teaching these classes all made clear lists of topics that should be learned in their class, and designed a standardized assessment of knowledge of those topics that could be given at the end of the semester year after year.
“Comparing performance on this test across semesters then allows you to infer differences in how well the students came to master the key topic areas laid out for the course,” says Rees-Jones. “Using this measure, we found that end-of-the-semester test scores declined by .2 standard deviations during spring 2019, which quantitatively is a pretty substantial decline.”
Second, they assessed whether certain groups of students were more affected by the pandemic.
“Specifically, we predicted student’s end-of-semester performance using information on whether they identified as an underrepresented minority, a female, a first-generation college attendee, or someone speaking English as a second language,” says Orlov. “While we did find evidence of some differences in performance across these groups, we did not find evidence that these differences changed during spring 2020.”
This suggests that, at least in the classes that were studied, according to Orlov, the negative effects of the pandemic were not particularly concentrated in one of these demographic groups.
“It would of course have very worrying equity implications if such differences were found,” he says.
And third, the study looked at whether the use of specific teaching methods resulted in a more successful transition to remote teaching.
Earlier research has shown repeatedly that students learn more when they actively work on problems either individually or together in the classroom relative to students who sit passively listening to a lecture and taking notes, according to McKee.
“We thought going into this project that these teaching methods could work especially well in this online-during-a-pandemic setting where students are more easily distracted and are hungry for social interaction,” he says. “So we were not surprised to find that students in classes with planned peer interaction scored significantly higher on our assessments.”
The study’s findings make the authors optimistic about future student learning outcomes even though many students and teachers remain in a period of substantial online instruction for three reasons, according to the authors of the study.
“First, online teaching experience seems to matter, and during 2020, many college faculty accumulated some experience,” says Rees-Jones.
“Second, we expected that disadvantaged groups would be further disadvantaged during the pandemic, but we found no statistical evidence of this concern,” says Orlov.
“Third, we have shown that it is possible to incorporate peer interaction or small group activities into synchronous online courses, and that it was significantly associated with improved learning, especially during the remotely taught portion of the semester,” says McKee.
