Thomas Edison and Nikola Tesla, famous inventors both, were also, famously, rivals. Their heated relationship came to a head in what became known as the “war of the currents.” Tesla favored alternating currents (AC) for the United States electrical system, Edison wanted direct currents, and, though AC ultimately won out, the rivalry never faded.
In the pursuit of science, such discord is not all that unusual; poking holes in the theory of the day is often how progress gets made. Yet, when neither party is willing to budge even a little, this kind of debate can stall scientific advancement. Penn’s Adversarial Collaboration Project wants to make sure that doesn’t happen.
Led by Cory Clark, a behavioral scientist and visiting scholar in the Department of Psychology, and in partnership with Penn Integrates Knowledge University Professor Philip Tetlock, the Adversarial Collaboration Project encourages scientists with competing perspectives to work together to design research that can adjudicate their dispute and test where the truth lies. Clark’s team is currently running 10 projects with several dozen researchers from some 30 institutions worldwide and recently published on this work in the Journal of Applied Research in Memory and Cognition.
“The current scientific model isn’t working. It’s often dysfunctional and unproductive and sometimes even counterproductive,” Clark says. “We make the argument that scholars who care about truth should participate in adversarial collaborations any time their own research contradicts the research of another scholar. My ultimate goal is to improve science as a tool for discovering the truth to help humans flourish.”
The history of adversarial collaborations
This idea of “adversarial collaboration” was first coined in the late 1990s or early 2000s by psychologist Daniel Kahneman, today an emeritus professor at Princeton. “He would be quick to say he didn’t invent the concept,” Tetlock says. “But he did come up with the phrase.”
In a 2003 American Psychologist paper, Kahneman wrote that he believed controversy “is a waste of effort” and that “doing angry science is a demeaning experience.” He sought an approach that didn’t pit one scientist against another but rather saw them working together in search of an answer.
“Adversarial collaboration involves a good-faith effort to conduct debates by carrying out joint research,” Kahneman wrote. “In some cases, an agreed-upon arbiter may be needed to lead the project and collect the data.” He practiced what he preached, engaging in several such scientific endeavors, including one with PIK Professor Barbara Mellers as arbiter.
Clark came to this work by way of research she’d been conducting on biases in science. She and Tetlock met at a conference in January 2021, where they both sat on a panel on political bias in psychology. “Up to that point, he and I had both worked separately on this topic,” Clark says.
Tetlock had also been a commentator on a recent paper of Clark’s, where he recommended adversarial collaboration as a potential solution for checking biases in science. “He had been a long-time proponent of adversarial collaborations but had never participated in one himself,” she says. “That summer, he and I started discussing ideas that would improve the truth-discovery process in science.” Tetlock suggested delving deeper into adversarial collaborations, and Clark ran with it. That resulted in the Adversarial Collaboration Project at Penn, housed in the School of Arts & Sciences.
Though more scholars now know about this type of scientific partnership, they’re still rare, Clark says. “People think it’s going to be a negative experience. We’re trying to demonstrate to our peers in the broader academic community that this is a viable, productive, rewarding way to do science.”
Projects underway
So far, several dozen researchers from universities in the U.S. and abroad have signed onto 10 studies underway by the Adversarial Collaboration Project. Each aims to answer a different question—whether the social sciences are politically biased, for example, or whether implicit bias predicts discrimination—and includes anywhere from three to 10 researchers.
The process of adversarial collaboration requires both parties to commit to what Clark calls the “conditions of falsifiability.” “You need to get everyone to say, ‘Here’s a study that we could actually run where it’s possible we could find A or it’s possible we could find B, and if we find B then I agree to at least slightly modify my current perspective.’”
Clark points to one about socially motivated reasoning, in which the two sides are trying to determine whether people evaluate new information in ways that confirm their preexisting beliefs. “In other words, will you decide that a new piece of information is higher quality if it supports what you want to believe rather than if it challenges your beliefs? People have been debating this for a long time,” Clark says.
The initial study brought together the “rationalist camp,” which states that people are decent at discovering the truth and evaluating information in an unbiased way, and the “motivated reasoning camp,” which argues that people are biased in analyzing new information and seek out data to support their beliefs. “In study 1, we found that people were equally sensitive to information quality when they didn’t want to believe something than when they didn’t care whether the information was true. But we also found that people evaluated undesirable information as lower quality,” Clark says. “So, both sides were a little bit right and a little bit wrong.” They’re designing a second study now, building on what they learned from the first.
Crucial to all of this is a neutral third party. “The ideal form of adversarial collaboration assumes that there’s a powerful centralized authority who can exert great pressure on the parties to be reasonable, to be good perspective-takers,” Tetlock says. “So far, Cory’s been playing a gentle diplomat or an intermediary who relies more on suasion.” Given the heated nature of the debates, Tetlock says it’s impressive just how much Clark and her teams have already accomplished. “I’ve been mostly gentle,” Clark admits. “To my delight, it’s been working so far.”
The method’s potential
Regardless of how Clark plays the role of arbiter, she believes this method has the potential for success in just about any field. “The behavioral sciences are where we’ve been focusing, but I think it could work in any discipline that’s collecting new data, with pretty much any empirical debate,” she says.
Tetlock agrees it has great promise, though he’s slightly more conservative, particularly about whether adversarial collaborations will work on disagreements with high policy stakes, where the players are “enmeshed with particular political constituencies.”
“It comes down to the question of, What is science? Science ceases to be science if the parties to the debate have asymmetric standards of proof, if they fear the error of falsely rejecting a hypothesis more or less than the error of accepting a false hypothesis—and if they reserve the right to move the goalposts of proof whenever convenient,” he says. “That’s difficult to resolve empirically. It’s a moral rather than a scientific commitment, and I’m not sure how you move through that.”
The nascent Adversarial Collaboration Project is still learning how to resolve such issues. What’s not up for debate, however, is what Clark says she hopes to accomplish with the work: to help researchers converge in their beliefs and move closer—and more quickly—to the truth. “I want scholars to approach their research questions with more curiosity and openness,” she says, “with more willingness to engage people who disagree with them, to be more like a detective trying to solve a mystery rather than defending a perspective.”
Cory Clark is executive director of the Adversarial Collaboration Project and a visiting scholar in the Department of Psychology in the School of Arts & Sciences at the University of Pennsylvania.
Philip Tetlock is the Leonore Annenberg Penn Integrates Knowledge University Professor of Democracy and Citizenship with appointments in the School of Arts & Sciences and the Wharton School.
This research was funded, in part, by the Searle Freedom Trust. Participants in the collaboration on socially motivated reasoning include Pete Ditto of the University of California, Irvine; Jon Haidt of New York University; Gordon Pennycook of the University of Regina; and Dave Rand of the Massachusetts Institute of Technology.
