Polling leading up to the 2020 election predicted a blue wave, with Democrats taking firm control of the Senate, picking up seats in the House and Joseph R. Biden winning the presidency by a landslide. But Republicans made gains in the House, and Democrats barely eked out a narrow control of the Senate after runoff elections in Georgia.
Penn’s Program on Opinion Research and Election Studies (PORES) has taken a leading role in a new collaborative report from the American Association of Public Opinion Researchers (AAPOR) that takes a look at what went wrong with polling in 2020.
Penn Today spoke with PORES faculty director John Lapinski, who is also the director of elections at NBC News and the Robert A. Fox Leadership Professor of Political Science in the School of Arts & Sciences, to get his take on the report and where the business of polling goes from here.
How did the report come to be, and what was the role of PORES?
Everybody was super worried about polling ahead of the presidential election because we’ve had problems with the polls for years. We thought we made some corrections after the 2016 election in how we weighted the data, and in 2018 it seemed to work. But 2020 obviously came up with some surprises. Across hundreds and hundreds of public and private polls, people thought that the Democrats were going to do better than they did. We had to ask ourselves what was going on.
I’m the director of elections at NBC News, and the person chairing the AAPOR report is Josh Clinton, a professor at Vanderbilt who also works for me on election nights at NBC News. Josh and I have written lots of papers together and work well together so I joined the team and helped him put together a 120-page report that assesses polling. PORES took a leading role in this. Our team collected a lot of the data for the report and assembled all the polls.
What was the conclusion?
What’s going to be unsatisfying with the report is that there are a lot of issues pollsters are facing. It’s not just one issue to fix.
There are issues that were specific to 2020 where you saw basically three different electorates: in-person Election Day voters; in-person early voters; and mail-in voters. Each group saw very different types of voters cast ballots using that specific method. So, in addition to getting the percentage of Trump support or Biden support for each type of vote right, pollsters also had to make some determinations about how large each group of voters was and how much of the total electorate each type of vote was going to make up in a specific state. You had to guess a little bit because you’re not sure exactly how many people are going to show up to vote in-person on Election Day. That was one issue.
Other issues are that response rates for polls are crazy low right now. That’s not necessarily a problem as long as the people that you’re getting are representative of the people that you want to respond to the poll. Clearly, the polling industry had problems with non-response. One of the big challenges that pollsters face today is reaching the type of voter who is much more likely to support Donald Trump. These people aren’t necessarily going to be strictly Republicans. They also could think of themselves as independents, and they generally have deep distrust in political institutions. That means that they have a distrust of polls, and they’re not going to take them. We’re trying to figure out now how we can work around that problem. We also don’t know what it is going to look like in 2022 and 2024. What’s a midterm versus a presidential year going to look like? What role is Donald Trump going to play in these upcoming elections? There are a lot of unknowns.
How does this get fixed for 2024?
We don’t know. The key thing is that there’s going to be certain types of polls that are not going to be as widely used like they were in the past. For example, some pollsters have used ‘random digit dial’ calls where you use statistical techniques to randomly call telephone numbers and reach people. There’s going to be more of a movement to call people off of voter lists because you know a lot about the people who you don’t reach. If you have a list of people who you’re calling, you might get some of them to answer your call and take the poll, but you won’t get others. However, you can use the voter registration list to start to figure out information about the people that you’re not getting, and you can see if you can make corrections based on that.
You mentioned that these Trump voters are very suspicious of political institutions. How do you regain the public’s trust, including among Democrats and people who typically trust polls?
I think the way polls are reported is going to have to be different. People put too much certainty in these polls. The aggregators don’t help with probabilities of wins. You could have a bunch of polls that are just a little off but all of them are pointing in the direction of Joe Biden winning, and then Nate Silver’s going to come up with a high probability that Biden’s going to win the overall election or win in a particular state. Obviously, Biden is the president; he did win. But people thought he was going to do a lot better in different places. There were some Senate seats that Democrats were pretty confident they were going to win. There was a lot of error. I talked to people in the private polling business, and the Democratic and Republican private polls were showing the same issues, too. It wasn’t as if somebody had some magic recipe; it was off everywhere across the board.
We’re going to have to be more careful. It might be that we report out different numbers; we could say that the plus or minus is a lot larger. What you see at times is a plus or minus three or four percentage points of error for example, but when we empirically look at the data and look at what those pluses and minuses were and how far off the polls were, in reality, the margin of error is often double. A lot of these polls that look like somebody has the lead, statistically they don't.
Now that the report is released, what are the next steps?
I think the next step will come down to individuals. A lot of people are going to be doing research. The National Election Pool, which is made up of NBC News, ABC News, CBS News, and CNN, is doing lots of experimental work to see how we can improve our polls. The Pew Research Center will be thinking about these issues as well. I think that everybody’s dipping their feet in the pool, figuring it out, and doing experiments.
The good news is most of these researchers will share the information or insights that they discover. Most people have an interest in trying to make sure that the polls overall are good. What we’re going to see is a lot of work leading into 2022. Some people are just closing their eyes, hoping for the best, and just continuing to do what they’ve done in the past. But that’s not good enough. The polls definitely have issues, and my team is going to go forward to see if we can identify what's going on. We recently did an experiment in the Texas 6th Congressional District’s special election where we tried different ways of contacting people to see if we could get better response rates, and we got different types of people taking the poll depending on the mode of contact. We mailed people requests to take surveys, we sent them text messages to invite them to take the poll, and we did traditional telephone surveys as well. We’re trying all these different things to see if somehow we can improve the accuracy of the numbers a little bit. We’ll have to do a number of these types of experiments, and once we’ve done them we’ll see whether or not we can make some improvements.
I can’t guarantee success on this one, and that’s a problem because polling is critical to our democracy. I think we can make polls better, but the question is, How good do they have to be for us to consider that a win?
