Sometimes the wronger a thing is, the harder it is to convince people that something is wrong. Hat seems to be the problem with scientific research. Numerous individual papers have come out over the years indicating that there are serious problems with the professional scientific establishment. How serious?
In 2005 John Ioannidis, an epidemiologist from Stanford University… argued, “most published research findings are probably false.”
I would say that “most published research findings are probably false” is pretty serious. But that’s just the problem: no one wants to believe that a problem could be that bad, and so there’s a kind of reflexive deafness whenever the topic comes up. Every time I’ve brought the issue up, usually with folks who I would consider suffer a severe case of scientism, I’m challenged to show the research. Conveniently, this article in The Economist, has done a great job of aggregating the most important studies on this topic over the past decade or so.
The fact that so much of our science literature is bad. The fact that maybe we can finally start to address the problem is good. Also, Weird Al.
2 thoughts on “Published Academic Research: Everything You Know Is Wrong”
Glad you brought this up. I’ve been in numerous discussions with hardcore “scientismists” where I’ve mentioned the apparently-deplorable state of academic research in criticizing allegedly “scientific findings” in various topics and every time I’m roundly and viciously criticized, if not ridiculed.
Several years ago when I was starting college I wrote a paper criticizing the state of some climate research for a biology class (this was prior to Mann and “Climategate”) and my professor nearly had an aneurysm. She was NOT pleased that I dared question not only the conclusions of the research, but the methodology and process of the research itself. My criticism didn’t stem from any particular delusions about my own expertise in the area of academic research, but from actual studies I’d read on the issue. I was prepared to consider the possibility that a lot, if not most, research was flawed, perhaps fundamentally. She, being invested in academia and research her entire life, was, quite simply, not.
Which, obviously, is the root of the problem in the first place.
“First, the statistics, which if perhaps off-putting are quite crucial. Scientists divide errors into two classes. A type I error is the mistake of thinking something is true when it is not (also known as a “false positive”). A type II error is thinking something is not true when in fact it is (a “false negative”). When testing a specific hypothesis, scientists run statistical checks to work out how likely it would be for data which seem to support the idea to have come about simply by chance. If the likelihood of such a false-positive conclusion is less than 5%, they deem the evidence that the hypothesis is true “statistically significant”. They are thus accepting that one result in 20 will be falsely positive—but one in 20 seems a satisfactorily low rate.”
I remember trying to explain this to a scientismist once on facebook. He insisted that that isn’t what hypothesis testing meant, I was using it all wrong, I don’t know anything, I’m just an idiot. I cited multiple academic web pages explaining how hypothesis tests work, and his rebuttal was to quote some random engineering paper that performed a hypothesis test to show me “how it is used in the literature”.
I even wrote a computer simulation to show him how confidence intervals (a specific kind of hypothesis test) fail roughly once every twenty times.
But it didn’t matter. Because science.
Anyway, thanks for posting.
Comments are closed.