galileo inquisiton painting

Galileo Faces the Roman Inquisition - Cristiano Banti
As the cultural edifices of western civilization are torn down one by one, there's one institution whose prestige and authority continues to grow - science. Respect for scientists has, in many quarters, been transformed into a form of worship. And questioning their authority akin to heresy. Yet this confidence is often misplaced. A disturbing number of scientists have been proven to be charlatans, their methods slipshod and their results bogus. Surveying numerous disciplines, Peter Shawn Taylor explains the implications of what's known as the "replication crisis" and reveals how science is trying to fix itself. Even non-scientists should be paying attention. — Peter Shawn Taylor

Yesterday it was social psychology. Today it's marine biology. Tomorrow, could it be Canadian history and the rest of society?

The sciences are currently grappling with a significant loss of credibility that has been called the "replication crisis." This crisis stems from a widening recognition that many published academic findings - including foundational work in some disciplines - cannot be replicated and, accordingly, cannot be independently verified. Both of these conditions are key elements of the modern scientific method. As a result, many well-cited scientific works long considered to reveal important truths are now considered unreliable or hopelessly biased. Some are the product of outright fraud. A recent example with a strong Canadian connection illustrates the problem.

Since 2009, numerous academic publications by marine biologists Philip Munday of Australia's James Cook University and Danielle Dixson of the University of Delaware have asserted that climate change will alter the behaviour of tropical fish in troubling ways. The pair's many experiments claimed to show that increased acidity in the oceans resulting from rising CO2 levels will cause various Pacific Ocean species to gradually lose their ability to flee predators and find their way home. The afflicted species include the orange clownfish, popular for its role in the Disney movie Finding Nemo.
marine biologists finding nemo Munday Dixson
© Bottom image: Disney Pixar
Nemo 911: Widely-cited work by marine biologists Philip Munday of James Cook University (top left) and Danielle Dixson of the University of Delaware (top right) claimed global warming would lead to significant behavioural changes for Pacific Ocean reef dwellers such as the orange clownfish, famous for its role in the Disney movie Finding Nemo.
Munday and Dixson said their striking findings entailed "direct consequences for...the sustainability of marine populations" around the world. Because this built on a familiar doom-and-gloom climate change narrative, these results were hailed as scientific fact. The duo's work appeared in a 2014 report of the UN's Intergovernmental Panel on Climate Change, which labelled it with "high confidence." The following year, Dixson gave a presentation to the White House. The looming disaster garnered further public attention due to the orange clownfish's pop culture appeal, leading to media headlines such as "Losing Nemo" and "Acid oceans turn 'Finding Nemo' fish deaf."

But was any of it true?

Intrigued by the calamitous results, in 2014 an international team with a large Canadian component assembled at the same Great Barrier Reef laboratory frequented by Munday and Dixson to try to understand the underlying biology behind these changes. "The behavioural effects in those original papers were some of the most phenomenally profound effects I'd ever seen in fish behaviour," recalls team member Graham Raby, a biology professor at Trent University in Peterborough. "Our intention was to build on that existing research and figure out the physiological mechanisms at work."

But despite repeated attempts at replicating Munday and Dixson's work, the team produced only null results. "We couldn't find any evidence of behavioural effects of ocean acidification," Raby says in an interview. Another team member, Dominique Roche, a researcher based at Carleton University in Ottawa and Université de Neuchâtel in Switzerland, notes that in hindsight the entire thesis never even made much sense. That's because at night coral reefs already emit levels of CO2 similar to what Munday and Dixson were warning about far into the future. "And the fish sleep inside the reefs," observes Roche. If rising levels of CO2 were going to change the behaviour of fish, it should already be evident.

After several years of meticulously documented research, in 2020 the seven-member team, including Raby and Roche, published a comprehensive refutation of the "Crazy Nemo" thesis in the prestigious science journal Nature. Some of the team went further and requested various international funding bodies investigate Munday and Dixson for academic misconduct since their work contained statistical anomalies generally associated with data fraud. "There are irregularities in the data that need to be investigated," states Roche firmly.

scientists refute global warming study
© Nature
The “cruel” truth: Munday and Dixson’s “Crazy Nemo” thesis was conclusively disproven in 2020 by a replication team that included Trent University biologist Graham Raby (top right) and Dominique Roche, a researcher at Carleton University and Université de Neuchâtel (bottom right).
The failed replication effort and calls for an investigation led to an explosion of controversy within the former clubby confines of marine biology. Munday and Dixson issued an angry refutation, declaring the replication study itself to be hopelessly flawed. An ally of the pair tweeted the accusation that "cruelty is the driving force of the work" of the replicators. Raby accepts this animosity as part of the process. "It's pretty common for the person whose work has been refuted to attack the replication effort," he shrugs.

Cruel or not, that a scientific finding enjoying broad public support and with significant public policy implications could not be repeated by independent researchers is a very serious matter, and a major blow to the entire field of marine biology. Everyone (save for the replication team) appeared so eager to accept a novel new story that bolstered popular beliefs about climate change that they failed to ask whether the results were even true. And when the findings proved unreliable, it was the truth-tellers who had to face a hostile crowd.

Despite the personal attacks and reputational damage suffered by marine biology, however, Raby firmly considers the refutation process to be "a good news story." Why? Because something that was mistakenly considered to be fact and widely publicized as such was eventually disproven through scrupulous adherence to the basic principles of the scientific method. By trying and failing to repeat the Crazy Nemo experiments, the team eliminated a falsehood and, in doing so, improved their discipline's knowledge base.

It is a clear demonstration of the vital process of conjecture and refutation. "As awareness of the replication crisis spreads across scientific fields, it is making them all more rigorous," says Raby optimistically. "This is an illustration of how science corrects itself. And science is the only method we have to find the right answers." Could such a commitment to skeptical inquiry fix the rest of society as well?

The Rise of the Replication Crisis

The phenomenon of the replication crisis is generally considered to have been first identified in a 2005 essay by Stanford University epidemiologist John P.A. Ioaniddis provocatively titled "Why Most Published Research Findings are False." Ioaniddis, well-known today as an outspoken critic of Covid-19 lockdowns and other pandemic policy mistakes, built his argument on the dubious incentive structure involved in the academic publishing process. Rather than revealing universal truths, Ioannidis figured much of what passes for academic research is simply an "accurate measure of the prevailing bias."
ioaniddis Nosak science replication crisis

Origin of a crisis: Stanford University epidemiologist John P.A. Ioaniddis (top left) provided a theoretical framework for the replication crisis in 2005; in 2015 University of Virginia psychologist Brian Nosek (bottom left) delivered the concrete proof.
Ioannidis' theoretical claims had little impact until a major scandal involving fraudulent psychology research in the Netherlands in 2011 created headlines around the world. At this point University of Virginia psychologist Brian Nosek decided to test the proposition that most published research was unreliable or biased. His ambitious approach was to assemble a team of 270 researchers who attempted to replicate 100 prominent, published psychology experiments. If the original results were legitimate and true, these studies should be easily repeatable. Nosek's ground-breaking investigation took four years to complete and revealed just 39 percent of the original studies could be fully validated - a stunning repudiation of what psychology had previously claimed to be fact.

Shocked by these results, Nosek and other academics quickly embarked on further tests of the reliability of published results in other social sciences. The results were similarly disappointing. Large swaths of original experiments could not be replicated, or the size of the claimed results proved far less than first reported. Since 2015 nearly every discipline in the natural and social sciences - from biomedicine to political science to economics to obstetrics to ophthalmology - has suffered a reputational blow as a result of disproven or unrepeatable results. Marine biology is just the most recent example of how far the rot has spread.

Ivan Oransky science retraction watch website

Nothing new here: According to Ivan Oransky, co-founder of the website Retraction Watch, the replication crisis in science is not a novel phenomenon; "it's just that people are now paying attention and we have the technology to expose it," he says.
Ivan Oransky is co-founder of Retraction Watch, the go-to source of information on the unfolding crisis; this is no exaggeration, as the website's database includes over 31,000 retractions of published results dating back to 2010. In an interview, Oransky says he dislikes the term "replication crisis", however. "It implies this is something new," he explains. "In fact, these problems have always been with us, it's just that people are now paying attention and we have the technology to expose it." According to Oransky's massive data bank, about two-thirds of retractions are the result of deliberate misconduct: data fraud, variable manipulation, plagiarism or some other form of academic bad behaviour.

A common theme among some of the highest-profile examples is that adherence to popular political objectives or storylines can often facilitate or provide cover for faulty research. The Crazy Nemo thesis' relationship to global warming is an obvious example.

Another stems from well-publicized research by American criminologist Eric Stewart of Florida State University, who claimed perceptions among white residents of black people were linked to historical evidence of racial violence. Stewart (who is black) said his polling data showed white people living in southern U.S. counties where lynchings had occurred in the past were more likely to consider black people a present-day threat and demand a harsher criminal justice system, than those living elsewhere. Such a finding obviously fits neatly with current claims of systemic racism. Yet Stewart's co-authors (who are white) later discovered the findings were based on phony poll results Stewart had supplied; the papers were soon retracted. Stewart responded to this scandal with the nasty, if predictable, accusation that his co-authors "essentially lynched me and my academic character."

In another example, this time from political science, published research by graduate student Michael LaCour at UCLA claimed it was possible to permanently alter the opinions of straight people about gay marriage by having gay canvassers talk to them at their front door. This was widely reported and enthusiastically hailed. It was subsequently revealed that LaCour had simply made up the data to satisfy his own interests. With few academic journals requiring that authors provide access to their raw empirical data, such deceptions can be despairingly easy to perpetrate.

Social Psychology: "Morality in an Experimental Idiom"

Perhaps no discipline has been hit harder by the replication crisis than social psychology. It featured prominently in Nosek's landmark 2015 study and now some of its most famous research is coming under similar scrutiny. Augustine Brannigan is professor emeritus of sociology at the University of Calgary and author of the 2021 book The Use and Misuse of the Experimental Method in Social Psychology: A Critical Examination of Classical Research. In it, Brannigan lays out numerous and convincing reasons to be deeply skeptical of some of the field's best-known research, including iconic studies such as Yale University professor Stanley Milgram's famous obedience experiment, the Stanford Prison Experiment and the Pygmalion Effect.

In the 1960s, Milgram claimed to show how ordinary people could be complicit in horrific crimes by having subjects administer progressively greater electric shocks to "students" during what they were told was a test of new learning methods. In reality, the "students" were actors only pretending to be shocked; the real test was whether the subjects would keep raising the shock level when told to do so by an authority figure. Milgram said his experiment was inspired by the trial of Nazi war criminal Adolf Eichmann, and offered insight into how ordinary German citizens willingly tolerated and even abetted the Holocaust.
Augustine Brannigan psychology debnk bad studies

University of Calgary professor emeritus of sociology Augustine Brannigan's 2021 book The Use and Misuse of the Experimental Method in Social Psychology: A Critical Examination of Classical Research takes a deeply skeptical look at some of social psychology's most influential experiments.
In a similar vein, the infamous Stanford Prison Experiment saw Stanford University psychologist Philip Zimbardo (a childhood friend of Milgram's) recruit students to participate as prisoners and guards in a mock prison he built in the basement of a campus building. The gusto with which some "guards" played their roles caused the experiment to rapidly disintegrate into abuse and rebellion, reactions Zimbardo later claimed provided vivid insights into the darker side of human nature, as well as prison policy.

Then there's the Pygmalion Effect arising from a series of experiments in the 1960s at Spruce Elementary School in San Francisco that purported to show how teachers graded students based on their own expectations and prejudices, rather than demonstrated ability. This was accomplished by tricking teachers into believing some of their students had been identified as having exceptional abilities. These findings, which carried strong racial implications, were central to the school busing debate in the U.S. in the 1970s and continue to wield significant influence over school policy today, especially since they appear to prove the pervasiveness of racism throughout the school system.
milgram prison experiment
© Left to right: Yale University Manuscripts and Archives, Stanford University Libraries, Courtesy of the South San Francisco Historical Society
Yale University psychologist Stanley Milgram's obedience experiment (left) reportedly explained the participation of ordinary Germans in the Holocaust, the Stanford Prison Experiment (middle) claimed to reveal dark secrets about human nature and the Pygmalion Effect, based on experiments at Spruce Elementary School in San Francisco (right), proposed that deceiving teachers about their students could improve their grades.
Brannigan's book identifies serious missteps and questions about the original research methodology in these and many other totemic works in social psychology that render them all unreproducible. Setting aside the fact that university ethics boards would never allow human test subjects to be treated in a similar fashion today, these fatal flaws include contradictory results that were deliberately suppressed and evidence the investigators manipulated their experiments to achieve a desired result. All this is in clear violation of the scientific method.

"They are all very entertaining studies, and they ask some really interesting questions," admits Brannigan. These dramatic "high-impact" experiments are also hugely influential, occupying large sections of undergraduate textbooks and representing the very foundations of the field. "But as science, they're terrible," Brannigan says. "Much of what passes for science in social psychology is just morality in an experimental idiom." Asked what such a revelation might mean for the future of the discipline, he retorts, "If the entire field were to disappear overnight, I don't think the world would be any worse for it."

Let a Thousand Replication Crises Bloom

If there's an overarching message arising from the replication crisis beyond the fate of social psychology, it's that relentlessly questioning all scientific work is the most effective cure for bad science. This includes scrutinizing new and flashy claims as soon as they are unveiled as well as re-evaluating long-accepted ideas that have already gained status as scientific certainty. Along with renewed emphasis on tough and unsentimental scientific replication must also come more rigorous fact-checking by scientific journals and a less chummy attitude towards the peer review process, with more emphasis on "review" and less on "peer."

"No science is immune to these challenges," says psychologist Nosek, who now heads up the Center for Open Science. Beyond requiring greater transparency in data and research methodologies, he says the best defence against phony or unreliable results is to ensure that skepticism is placed at the centre of all scientific inquiry. "If we don't have a system that is self-skeptical and questions existing findings, then we will create a world that is built, more and more, on false confidence," he says in an interview. Everything must be tested and tested again, regardless of how much that may threaten existing beliefs or comfortable political viewpoints. "Even the replications need to be replicated," Nosek states. "It all needs to be treated with skepticism."

Patrice dutil historian replication crisis
© Ryerson University
Why restrict the replication crisis to the sciences? Historian and political scientist Patrice Dutil of Toronto's Ryerson University admits that while the humanities generally lack replicable experiments, academic skepticism and unfettered debate are still crucial to good scholarship.
Could such an attitude improve disciplines outside the sciences as well?

Patrice Dutil teaches public administration and political science at Toronto's Ryerson University, although much of his academic research involves Canadian history. He worries the humanities are currently suffering from the same ideological myopia and bad research habits that characterize the replication crisis. Of course, history differs in many significant ways from sciences such as social psychology or marine biology, Dutil admits. There are no experiments to replicate or reams of data to test, and much of the published work consists of subjective interpretations of readily available evidence. "Beyond ensuring our footnotes are in order, there aren't a lot of numbers to crunch," Dutil notes wryly. Nonetheless, he argues, skepticism remains critical to good scholarship.

Dutil points to his own institution's plan for a name change as a prime example of the need for history to experience its own replication crisis. A report from a school committee recently recommended that 19th century educator Egerton Ryerson's name be removed from the school due to his alleged role in creating Canada's Indian Residential School system, a demand readily accepted by school leadership. "This entire episode represents a wholesale misrepresentation of historical facts," Dutil says. "Nothing the [committee's] report says is verifiable. In fact, its interpretations of Ryerson are plain wrong."

According to Dutil, "Ryerson is being 'hanged' because of a five-page letter," referring to a brief report Ryerson wrote in 1847 sketching out a proposed national schooling system for natives. Most of this short document is based on Ryerson's observations about a philanthropic school for poor children in Hofwyl, Switzerland. It would be another four decades before the federal government created a comprehensive national system for native education — and it looked nothing like the Hofwyl school.
ryerson statue torn down canada
© Right photo: The Canadian Press/Chris Young
Before and after bad history: Dutil points to the recent treatment of his school's namesake, 19th century educator Egerton Ryerson, as an example of how ideological narratives have triumphed over scrupulous research and verifiable facts.
"To hold Ryerson as the engineer of Canada's Indian Residential Schools on this basis is a complete denial of logic," Dutil charges. It is a claim fuelled by popular political narratives about anti-colonialism and white supremacy rather than scrupulous, replicable research. Such an abuse of historical evidence, he argues, should be considered as big a scandal as those that have rocked marine biology, social psychology or criminology.

In another example of how Canadian history is plagued by unverifiable claims, Dutil points to the July 1 statement from the Canadian Historical Association claiming a "broad consensus" for the notion that this country is guilty of genocide in its policies towards Indigenous people. Borrowing from the language of the replication crisis, Dutil argues, "This is emphatically not a settled issue. Such a claim would have to rest on verifiable evidence, and that does not exist. To say such a thing is true is a gross abuse of authority."

Yet the genocide story is quickly becoming an unquestioned - and almost unquestionable - belief across Canadian society. In British Columbia, for example, social media commentator Aaron Gunn saw his plans to run for the leadership of the B.C. Liberal Party stymied by a party committee that claimed his candidacy would be "inconsistent" with the party's values. As evidence, the committee cited Twitter posts addressing Indigenous issues in which Gunn complained about the use of "a loaded word like 'genocide' that doesn't remotely reflect the reality of what happened."

Anyone who declares themselves skeptical of claims of a Canadian "genocide" is now at risk of cancellation. But once any topic is declared off-limits in this way, it becomes impossible to test its underlying assumptions and claims. Such a closed system inevitably gives shelter to falsehoods and misinformation. Despite the important lessons of the replication crisis, skeptical inquiry is actually becoming increasingly rarer outside the sciences.

The Need for Universal Skepticism

In 2018 Dutch academics Rik Peels and Lex Bouter proposed the "desirability of replication in the humanities," and sketched out a system by which research in politics, history, linguistics or literature could be subjected to the same rigorous efforts now familiar to the sciences in order to weed out phony or unreliable claims. If their goal was to start a movement, however, Peels and Bouter appear to have failed.

And the pandemic is making things worse. In a recent essay for the online magazine Knowable, entitled "Question the 'lab leak' theory. But don't call it a conspiracy," Retraction Watch's Oransky notes that honest skepticism has often been dismissed and discounted as partisan posturing by political opponents throughout the Covid-19 era. At a moment when we should be honouring the scientific method, we are instead experiencing an unscientific rush to judgement across many important issues.
Wuhan lab conspiracy bad science
© AP Photo/Ng Han Guan
Reductio Ad Conspiracy Theorem: Throughout the pandemic, proponents of honest skepticism regarding many important issues, including the potential role of the Wuhan Institute of Virology (pictured) in the release of Covid-19 virus, have been dismissed as conspiracy theorists or partisan cranks, writes Oransky.
As an example, Oransky points out that anyone prepared to consider whether the Covid-19 virus first spread from the Wuhan Institute of Virology in China was swiftly labelled as a "conspiracy theorist." But such a kneejerk response is an impediment to discovering the truth. The same goes for debates over the side-effects of vaccines or the value of vaccine mandates. Conjecture and refutation cannot operate in an environment that ring-fences existing biases. "It is good to ask questions, and it is good to doubt," Oransky writes. From his Knowable essay:
"In the last 18 months...practically no one could have a dispassionate discussion of the evidence - or lack thereof - for using various older drugs, usually approved to treat parasites, against Covid-19. Such discussion frequently escalated into screaming matches between polarized opposites. If you ever made a comment that could be interpreted as saying that Donald Trump was not wrong about absolutely everything - including his boosting of one such drug, hydroxychloroquine - you were clearly promoting the drug because you were partisan. If you dismissed the drug, surely you were a partisan on the other side."
Polarization of this sort is clearly inimical to the scientific method - and, indeed, to all forms of research and open intellectual inquiry - as it presupposes the answers before the questions can be asked and any claims tested. The best solution in all cases is to simply let open inquiry do its work. And until it supplies a reliable answer, it is foolish and dangerous to limit debate or attack honest skepticism.
Galileo before the Holy Office vatican painting
© Joseph-Nicolas Robert-Fleury
Without acceptance of the virtues of honest skepticism, it is impossible to push forward the boundaries of human knowledge or disprove accepted falsehoods; 17th century astronomer Galileo Galilei’s appearance before the Roman Inquisition, in which he was forced to recant his (essentially correct) theory of planetary movement because it conflicted with church dogma.
If there is a caveat to the skeptic's crucial role in the scientific method, however, it's in making a distinction between skepticism and cynicism. "How do we encourage skepticism and the process of questioning assumptions and results in a way that does not turn into weaponized cynicism?" Oransky asks.

Drawing this line is important if we want to bring clarity to important public policy issues. Healthy skepticism seeks to test what we think we know in order to improve confidence in the facts we must rely upon - and to root out malfeasance, incompetence and fraud. Cynicism, on the other hand, tends to disregard all knowledge and expertise as a matter of habit. At a time when absolute certainty is very much in vogue and any dissent treated with disdain - on charged topics as varied as gender identity and biology, racism, meritocracy, addiction, criminal justice, sexual violence and all the other fault lines that define our current cultural revolution - we need more of the former and less of the latter. It should be possible to raise important questions without being accused of being cynically anti-science or threatening public order.

"A return to scientific, or sobering, skepticism is necessary," Brannigan says. "We must learn to be suspicious of all findings generally, and to be even more suspicious of findings that seem too good to be true, or blow in the right directions." As the replication crisis has shown, dearly-held scientific facts can often turn out to be unscientific myths when subjected to the appropriate scrutiny. But why should such a standard be reserved for the sciences alone? If we seek to know what is true, every academic discipline and area of public policy debate should bear the same burden of proof. Take nothing for granted.
Peter Shawn Taylor is senior features editor of C2C Journal. He lives in Waterloo, Ontario.