Are Most Published Scientific Research Findings False?
/Readers interested in the subject of science versus consensus and orthodoxy enforcement might enjoy the article "Broken Science" by Ronald Bailey appearing in the current (February 2016) print edition of Reason Magazine. (Here is the link for reason.com. The article doesn't seem to be at the online site yet, although I assume it will show up within a few days.)
Bailey delves in some depth into the subject of the remarkable amount of published scientific research that cannot later be replicated and ultimately turns out to be wrong. How much? As much as half or more. Citing extensive evidence of pervasive failed replication and/or complete falsification of previously published research results, Bailey reasonably asks whether "science" is "broken."
If you have read my post from a few days ago on the scientific method, you know my answer. No, this does not indicate that science is broken. It is in the nature of science that huge numbers of hypotheses that seem brilliant and reasonable and intuitive and obviously true turn out to be false. Science is not a process of "proof" of hypotheses, but rather a process of sequential falsification of some hypotheses in favor of better hypotheses. Publication is just an indication that a hypothesis has survived an initial round of testing, inevitably by the very people who posed the hypothesis in the first place and therefore have a strong interest that the hypothesis should turn out to be right. Even after initial publication of favorable results, every hypothesis should still be taken with a huge grain of salt; and, after multiple independent replications, a hypothesis should still be taken with a somewhat diminished but still substantial grain of salt. Eventually most hypotheses -- and, given enough time, maybe all -- will fall, in some cases to a slightly improved hypothesis, in other cases to a completely different hypothesis, and in many cases to the dreaded "null hypothesis."
Bailey begins by citing the famous 2005 PLOS Medicine article by Stanford statistician John Ioannidis titled "Why Most Published Research Findings Are False."
Ioannidis showed, for instance, that about one-third of the results of highly cited original clinical research studies were shown to be wrong or exaggerated by subsequent research. "For many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias," he argued. Today, he says science is still wracked by the reproducibility problem: "In several fields, it is likely that most published research is still false."
After that article was published, many criticized Ioannidis for exaggerating his own results; but Bailey cites multiple subsequent efforts that have repeatedly shown stunning rates of non-replicability of published results:
In 2012, researchers at the pharmaceutical company Amgen reported in Nature that they were able to replicate the findings of only six out of 53 (11 percent) landmark published preclinical cancer studies. . . . In 2011, researchers at Bayer Healthcare reported that they could not replicate 43 of the 67 published preclinical studies that the company had been relying on to develop cancer and cardiovascular treatments and diagnostics. . . . Ioannidis estimates that "in biomedical sciences, non-replication rates that have been described range from more than 90 percent for observational associations (e.g., nutrient X causes cancer Y), to 75-90 percent for preclinical research (trying to find new drug targets)" . . . . In August, Science reported that only one-third of 100 psychological studies published in three leading psychology journals could be adequately replicated. In October, a panel of senior researchers convened by the British Academy of Medical Sciences (BAMS) issued a major report on research reproducibility indicating that the false discovery rate in some areas of biomedicine could be as high as 69 percent.
If you ask me, all of this is totally normal and to be expected. This is not an indication that science is "broken," but is rather the very nature of how science works. I just have two questions: (1) Why does the news service in the elevator at my office building in Manhattan breathlessly report literally every newly-published piece of research in the field of bio-medicine as if it is likely to be true (e.g., people who drink green tea are 17% less likely to develop colon cancer!! people who eat tomatoes are 23% less likely to get breast cancer!! etc., etc.)? Don't they know that almost all of this stuff will ultimately prove to be wrong? and (2) Why is the field of climate science immune to the process of hypothesis falsification that is the essence of the scientific method in all other fields claiming the mantle of the term "science"?