mead cohen berger shevtsova garfinkle michta grygiel blankenhorn
"Settled" Science?

The British Guardian newspaper reports that an elementary statistical error has been found in half the published scientific papers in which the error could be made.  Examining published articles in prestigious, peer-reviewed neuroscience journals, researchers found that a howling error was widely and routinely made and accepted.

How often? Nieuwenhuis looked at 513 papers published in five prestigious neuroscience journals over two years. In half the 157 studies where this error could have been made, it was. They broadened their search to 120 cellular and molecular articles in Nature Neuroscience, during 2009 and 2010: they found 25 studies committing this fallacy, and not one single paper analysed differences in effect sizes correctly.

The error would lead scientists to mistake whether an experimental result was significant or not and it amounts to a systematic bias that makes experimental results look more important than they are.

The article does not say, and Via Meadia does not know, how widespread basic statistical ignorance is among scientists in other fields.  But if anyone is wondering why laypeople are less trusting of “settled science” than they used to be, stories like this one suggest an explanation.

Features Icon
show comments
  • Sean

    Scientist are humans too, and should not be considered beyond questioning. People forget or do not know that it was science and scientists that created the hierarchy of races in 18th, 19th, and 20th centuries. These presumed hierarchies lent scientific credence to racism and contributed to some of the most despicable crimes in modern human including slavery and the holocaust. I write this not to disparage the scientific field or scientists, but to point out that science is neither perfect nor always correct. Sometimes legitimate mistakes are made during research, as Dr. Mead relates in this article, and sometimes mistakes are made out of bias.

  • Jim.

    Worst are the sciences that claim that stochastic models can be used to prove anything.

    I’ve worked with stochastic models, extended Kalman filters and batch-fit data processing specifically. While programming them is a science, running them and getting good data out is an art.

    You fiddle with some numbers, then check to see whether they line up with historical data more or less. After a certain number of iterations you declare it “done” depending on how much time you have to spend on the run, and whether the numbers are sufficient for whatever application you’re using them for.

    Can this go wrong? Absolutely. This analytical method gives you error bars, but those aren’t magically perfect; in a bad run, they’re as likely to be wrong as the estimate itself.

    And there’s no particularly good way to check to see if you’ve had a bad run, apart from matching to historical data. You can propagate values into the future, but that’s not reliable, period.

    Putting in the most accurate and precise model you can does NOT, I repeat does NOT lead you to the best results. Errors, from egregious modeling mistakes (like getting a sign wrong in a critical equation, something that’s happened in my group) to oversimplified models, can lead to BETTER matching of historical data. Put in the hyper-realistic model, and your filter is just as likely, if not more likely, to wander off with its own ideas of reality and never come back to anything resembling actual incoming data. (A situation technically known as “smugness”.)

    On the other hand, in nonlinear (chaotic) systems, which are tremendously sensitive to initial conditions / small perturbations / what have you, oversimplified modeling has been proven to lead to huge errors in a surprisingly short timescale.

    In other words… [darned] if you do, [darned] if you don’t.

    Some people look at the scientific method and mistakenly believe that it implies that everything is knowable. It’s not. Science is a powerful tool, to be sure. Computers can dramatically reduce the time it takes to do calculations. Statistical methods can often give you good estimates, and give you insight you might not otherwise have had.

    But NONE of it gives you the sort of certainty that many people claim (AGW advocates spring to mind, here). Errors creep in with computers, and flood in, only marginally controlled, with statistical methods.

    Skepticism of any conclusion based on stochastic (statistical) methods is fully justified.

  • Joe Citizen

    The phrase ““Settled” Science” of course, is used almost exclusively in public discourse to refer to the claim that anthropogenic climate change is a real problem that needs to be addressed.

    You think that noting the existence of errors in psychology papers can undermine global warming science???

    It does seem that you are trying to deceive your readers in that manner, given your lede:
    “…error has been found in half the published scientific papers in which the error could be made.”

    A statement which is only true once you qualify it by mentioning that you are referring to a small pool of psychological papers, not all scientific papers.

    Are you a hack or something?

  • Barbara Piper

    Prof. Mead draws our attention to a nice study from which one may draw many conclusions. My husband, a physician who has conducted a number of studies of drug effectiveness, tells me that the kind of technical statistical error discussed in the study that Prof. Mead cites is one that drug research is especially cautious about, since FDA approval often requires not merely that a new drug produce the hoped-for effects, but that it does so better than existing drugs, and this difference between differences is exactly what needs to be assessed. His impression was that the complex science involved in much of research mentioned requires deep knowledge of biology, chemistry, lab techniques, etc., and the statistics can be weak without the participation of professional statisticians in the research.

    I was interested, myself, in Prof. Mead’s suggestion that stories such as the one he cites may be one reason why the lay public is less trusting of “settled science” “than they used to be”. My impression has been that public trust in science comes and goes, and has done so since American colonial times. We have passed through a period of significant trust, even admiration for science since the Second World War and the Manhattan Project proved the value of intensely focused science on a single urgent problem, and then a decade later the response to Sputnik. We even persuaded women to stop breast-feeding in the 1950s and adopt the scientific method of infant formulas and sterile bottles. The American public has backed off from that consensus of support for ‘science’ over the past few years, but perhaps we have only returned to a level of suspicion and distrust that we have seen in other eras of American history.

    One important early wave of skepticism about science was part of the Jacksonian ideals of egalitarianism that extended from the colonial 18th century to about 1850, when the public was deeply suspicious of what we would now call scientific experts, (and medicine and law), believing that such claims to special authority were suspiciously European, elitist, and contrary to new American ideals. In the 1830s and 40s states abolished most of their laws requiring licensure to practice medicine or law, pressured by the democratic sentiment that anyone should be able to learn and practice those fields without some self-appointed group of “experts” deciding who could or could not. Restrictive licensure was only supported by those elite interest groups to begin with. Science rose in public esteem after the 1850s, but mainly because of the newly emerging professional class and its adoption of science as a kind of interest or hobby. Suspicions of science once again arose after the 1890s, and it appears that only physics remained trustworthy for the next 40 years or so, perhaps because physicists were publically re-inventing their field, and the revolutionary pains were honest, even if the new physics was bewildering.

    Sorry for the long ramble. In short, I wonder if the article Prof. Mead cites is a symptom of our current attitudes rather than a cause of those attitudes. Perhaps a sociologist of science has some thoughts on this?

  • Luke Lea

    Basic statistical reasoning should replace calculus as the go-to course after algebra. Calc is sooo . . 19th century! Probability rules! Literally.

  • clazy8

    Luke Lea’s proposal is spot on. How often has calculus been useful to me in the last 30 years? Very seldom, although I am not an engineer or physicist. Statistics, on the other hand — and the underlying theory of probability — these things turn up every other day, whether we are trying to judge scientific claims or only making personal decisions based on uncertain facts. A basic understanding of probability is fundamental to clear thinking.

  • clazy8

    As Joe Citizen notes, the term “settled science” is seldom used in any context other than AGW, but he neglects to consider what that suggests about the very notion of “settled science”: it is not useful except as political rhetoric. Indeed, it is practically oxymoronic. Some questions are more “settled” than others, to be sure, but only within the constraints of specific assumptions. Change assumptions, and the answers collapse beneath proliferating complexities. “Settled science” is nothing but a screen used to obscure the myriad of tenuous assumptions behind the apocalyptic predictions of AGW theory.

    The progress of science is limited by the questions that are asked and the tools available to answer them. Sean’s example of racial hierarchies shows what can happen when the tools are crude and the questions unclear: assumptions become answers. In the 19th century, scientists had only the most basic understanding of the factors behind heredity and natural selection. Limited in what they could know, and oblivious to the extent of their ignorance, they were comfortable making assertions that were based on little evidence, but which confirmed the common wisdom.

    What happens when such answers, dressed in the authority of science, coincide with utopian social ideals? You get the eugenics movement. Scientists, activists and politicians eager to improve the lot of humanity believed science and good intentions justified sterilization of people judged mentally or physically defective. The truth of their position was so obvious that over 30 US states adopted related laws and engaged in the forced sterilization of tens of thousands of citizens during the first several decades of the 20th century.

    The proponents of AGW mandates are utopians like the eugenicists, but whereas the latter would purify humanity, the former would purify the earth of humanity’s destructive influence. Instead of forced sterilization, today’s utopians would forcibly constrain economic freedom, even if it would wreck the global economy. They would do this in spite of the fact that AGW theory has predicted nothing accurately, and climate science in general has only the crudest understanding of its subject. I suspect that in fifty years, we will look back on AGW activism in the same way we now look back on eugenics, a horrifying perversion of science.

  • megapotamus

    Lighten up! It ain’t brain surgery!

    Yes, this certainly IS indicting of catastrophic anthropogenic global warming as an exemplar. “Peer review”, like the old KGB is both the sword and the shield of those who would forbid fire from the knowledge of man. Look into your alleged “facts” Joe. Ever heard of Climategate? This was the precursor to wikileaks. The most prominent leaks were those that showed the elites of Warmology happily and openly (to each other) engaging in fraud. Lying, in other words. Altering their data. And not incidentally slandering anyone who asked for said data and even destroying it. Is that the sum of it? Aw, [heck] naw! You have cherry picking, log rolling, sex-selection…. any scheme you could concoct in fiction attributable to the Koch Bros has been practiced by these scientologists as a matter of public record. Science, this ain’t. ‘t ain’t settled neither.

© The American Interest LLC 2005-2016 About Us Masthead Submissions Advertise Customer Service