the end wall
© Crawford Jolly Unsplash
Science writer Tibi Puiu reports on new findings that reflect what many today, have begun to suspect:
"Over the past few decades, the number of science and technology research papers published has soared, rising at a rate of nearly 10% each year. In the biomedical field alone, there are more than a million papers pouring into the PubMed database each year, or around two studies per minute...

The new study revealed that the "disruptiveness" of contemporary science has decreased, rendering ever diminishing returns. In this particular context, authors define disruptiveness as the degree to which a study departs from previous literature and renders it obsolete. In other words, a highly disruptive study is one that completely changes the way we think about a particular topic and renders previous research on the subject obsolete."

Similarly, Kelsey Piper at Vox tells readers of her Future Perfect newsletter, despite a more than tenfold inflation-adjusted increase in U.S.-government spending on science โ€” and that of many other sources โ€” since 1955, "It feels like we're doing more research and getting less out of it."

Whence the Pessimism?

The pessimism comes through an authoritative source: An open-access paper in the world's pre-eminent science journal, Nature. From the Abstract:
Recent decades have witnessed exponential growth in the volume of new scientific and technological knowledge, thereby creating conditions that should be ripe for major advances. Yet contrary to this view, studies suggest that progress is slowing in several major fields. Here, we analyse these claims at scale across six decades, using data on 45 million papers and 3.9 million patents from six large-scale datasets, together with a new quantitative metric โ€” the CD index12 โ€” that characterizes how papers and patents change networks of citations in science and technology.

We find that papers and patents are increasingly less likely to break with the past in ways that push science and technology in new directions. This pattern holds universally across fields and is robust across multiple different citation- and text-based metrics.

Puiu explains,
Researchers at the University of Minnesota's Carlson School of Management used citation data from 45 million scientific papers and 3.9 US-based patents to calculate a "CD index", a measure of disruption which ranged from -1 for the least disruptive work to 1 for the most disruptive. The CD index is basically a proxy for innovation and is computed by looking at the number and quality of citations a paper or patent received five years after publication, assuming that the more disruptive a study is, the less its predecessors would be cited because it is, by then, considered outdated knowledge.

The authors were stunned to learn that the average CD index declined by more than 90% between 1945 and 2010 for research manuscripts, and by more than 78% from 1980 to 2010 for patents. This decline in disruptiveness was observed in all fields and patent types, even when taking into account potential differences in citation practices.

What these figures mean is that research papers and patents have become increasingly conservative, consolidating or building upon previous knowledge, rather than breaking new ground.

John Horgan's Warning

Back in 1996, prominent science writer John Horgan warned about this very thing in The End of Science: Facing the Limits of Knowledge in the Twilight of the Scientific Age. But he makes clear, as he explained in considerable detail at The Edge in 1997, that in his view, the current materialist paradigm is the answer to all our questions:
"... given how far science has already come, and given the limits constraining further research, science will be hard-pressed to make any truly profound additions to the knowledge it has already generated. Further research may yield no more great revelations or revolutions but only incremental returns."
His reasoning was clear enough:
"... evolutionary biology keeps reminding us that we are animals, designed by natural selection not for discovering deep truths of nature but for breeding."
Science is winding down because there is little more for mere breeding animals to discover.

Needless to say, it was not a popular thesis. Not, of course, on account of the materialism; the problem was rather the implication that materialism inevitably limits possible knowledge. In 2015, he confirmed that view at Scientific American:
"My book has now sustained almost two decades worth of attacks, some triggered by genuine scientific advances, from the completion of the Human Genome Project to the discovery of the Higgs boson. So do I take anything back? Hell no."
But materialism (nature is all there is) can present one of two types of problems. It can be an absolute limit (because it is true) or it can be a self-imposed one (because it is assumed but is not true). In the latter case, the many problems cited today as holding back science are not signs of an approaching absolute limit but simply choices that research establishments have made โ€” choices they are free to reverse, or can be pressured to reverse. Thus change is possible and might lead to continued great discoveries.

One Scientist's Hard Take

Experimental physicist Rob Sheldon offered a personal reflection on what has changed over his long career:
I've been blowing that horn for almost 10 years now.

Science is most definitely going through an existential crisis. I have written papers or proposals in a half-dozen fields, and nearly all of them are in some sort of stasis or doldrums.

In the field of Space Physics, a 1994 National Academy of Science publication, A Space Physics Paradox: Why Has Increased Funding Been Accompanied by Decreased Effectiveness in the Conduct of Space Physics Research?, basically made the author, who was director of Space Science Lab at NOAA Boulder, a pariah among the NASA bureaucracy.

Now this publication came about 10 years after the effectiveness had diminished, so by the mid-1980s it was already being noticed. Then in 1999 or thereabouts, a retired director of a Max Planck Institute, Ian Axford, was hired by UA Huntsville, and I got a chance to ask him these questions:

He had lived through the heyday of the 1960s space race between Russia and US, when money and rockets were thrown at space physics if it would enable us to beat Russia in space. So he had seen the rise and fall of the field. His comment to me was that space physics was now seen as a "cottage industry", keeping some tenured scientists afloat, but it was no longer seen as worth investing in.

As if to validate his prediction, the last "magnetospheric" satellite and the one I wrote my PhD thesis on, the AMPTE satellite, was launched in 1984. But it would not be until 2012 that a similar satellite was launched, ostensibly for the same purposes ("understanding the Earth's radiation belts"), but implicitly to keep all those middle-aged scientists (my cohort) fed after their advisors retired.

In 1994, I read the booklet and concurred that this was unique to my field. But as I started roaming the hallways, searching for active research groups, I realized that it was true in all the related fields: X-ray astronomy, gamma-ray astronomy, lunar geology, plasma physics modelling, rocketry.

I remember a conversation I had with the local NASA director, Jim Spann, that space physics needed to follow the non-conventional model of the global hydrology group headed by John Christy and Roy Spencer, by going against the flow. His reaction was swift and angry. They had rejected NASA and we would never follow their lead.

Slowly it dawned on me, that NASA had been captured by politics, and politics determined the grants and papers and satellites in all these fields. The doldrums experienced in all the fields I had looked into were essentially political. The fruitless search for The Universe has dark matter could be tied to federal funding. The failure to build a better rocket (despite Von Braun's pioneering nuclear rocket work in 1972) was tied to federal funding.

The fact that Elon Musk has been so successful in building rockets and satellites is that he has bypassed federal controls. Fields that have not been captured by federal agencies were flourishing, but they were becoming rare.

As to the sociology of science, the 20th century was a century of change. When Einstein attended the Solvay conference in 1927, everyone there was a disruptor, an innovator. And everyone there did his science without government funding. Today, it would be hard to find a single scientist in a conference of 1000 who had not received government funding. And with the funding come the strings that hold back disruptors. There are undoubtedly many other factors contributing โ€” the rise of the research university, the arrival of peer review, the publish or perish criteria, the sheer number of career scientists, the exponential rise in publications โ€” but in my mind, it is the government control that is the dominating factor.
It sounds as though Sheldon believes that change may be possible in principle. After all, Elon Musk โ€” a classic disruptor โ€” is doing what huge, powerful bureaucracies simply can't.

So the issue isn't that it can't be done.