You May Now Ignore All Scientific Studies

Readers, I've developed a cognitive rule of thumb for "scientific studies" that I now use whenever I hear or read about any study. Here it is:

Heuristic: Any study you see, ever, anywhere, that happens to reach you through the media is wrong in some fundamentally significant way. You may safely ignore it.

My first halting step toward this admittedly cynical mental rule came a long time ago: when "scientists" decided--and later undecided--that margarine was better for you than butter. I took another halting step toward this cynical rule when the "don't eat too many eggs" study came out, a study that blissfully ignored the fact that dietary cholesterol is not blood serum cholesterol.

My steps became a lot less halting thereafter, as I thought through what "scientists" used to think was true. Things like:

* "Healthy whole grains."
* An entirely upside down food pyramid.
* Recommendations for statin meds because of a "studies show" link to cardiovascular health that turned out to be wholly imaginary.
* The fact that the medical profession doesn't seem to know what normal blood pressure is.
* That toast causes cancer.

There are many, many more examples, of course, some amusing, some not funny at all, some actually life-or-death. And all wrong. And if you have even a cursory understanding of decline effects[1] and the great crisis of reproducibility,[2] you will lose any remaining faith in "studies show" science.

Still more severe examples include South Korea's misguided war on thyroid cancer[3], or the highly counterintuitive discoveries that annual mammography screenings have zero effect on life expectancy and that, shockingly, prostate cancer screenings negatively affect life quality and life expectancy.[4]

And when it seems "studies show" science couldn't be any more wronger, it gets worse: We've discovered that even some of the most important and foundational studies in twentieth century psychology cannot be reproduced.[5] We're seeing major, domain-shattering acts of sheer data fabrication--see vaccines[6] and social psychology[7] for two object examples. Even in the food industry, one of America's best known dietary scientists, Cornell University's Brian Wansink, just got caught torturing his data to find links that don't really exist.[8]

And this is to say nothing of the various types of errors that inevitably show up in the study industry, as well as the rampant errors common to media coverage trumpeting study findings. These include errors like baseline risk error (an increase in a risk that's too small to matter is still too small to matter), p-hacking (mining data for statistical anomalies first, then forming a post hoc hypothesis on something that is almost assuredly spurious), and many others, some of which we've already discussed elsewhere at Casual Kitchen. And don't forget the well-known pressure to "publish or perish" in academia, which is most likely the prime driver of many of the industry's data fabrication and data-torturing scandals.

Last but certainly not least, there's the structural fact that the media--and it does not matter which media--inevitably oversimplifies or exaggerates all study claims to the point of making them into anti-information. And I have yet to read an article in my life saying "XYZ perfectly normal everyday activity shows no link to cancer." It's always the opposite.

You can safely ignore it all.

Finally, for any readers who consider this article to be somehow anti-science, keep in mind: "studies show" science is not science. It never was.



Footnotes/Resources:
[1] A readable article on the "Decline Effect," a genre of the problem of reproducibility.

[2] More good articles on the reproducibility crisis here and here.

[3] Korea and its misguided search for thyroid cancer as a textbook example of two types of silent risks: overscreening risk and overdiagnosis risk.

[4] For excellent discussions of why prostate screenings and mammography screenings carry silent risks, see Gerd Giggerenzer's book Risk Savvy and Gilbert Welch's book Less Medicine, More Health.

[5] Classic and foundational studies in pyschology cannot be successfully replicated.

[6] See for example the famously fraudulent “vaccines cause autism” study. Ironically, the paper pointing out the fraudulence itself had to be later corrected because it never acknowledged financial support from MMR vaccine makers. (!)

[7] “2011: A Year of Horrors” in social psychology.

[8] Bombshell allegations of data mining, p-hacking and other types of statistical data torture shatter the credibility of dietary science professor Brain Wansink's entire department at Cornell University: here, here, and most depressingly: here.


3 comments:

JCS said...

Nice summary. Science is not a straight line to the truth. Implementation is subject to all of the flaws of human nature. It takes decades and an immense amount of independent research to tease out low correlation effects.

Science also suffers for two things: an overeducated public, and an undereducated public. The former has been on top of scientific studies far too early in the process. Without reproducability, the public shouldn't even hear about it. The undereducated part is that they (we?) don't know that we need to look at meta-studies and understand statistics- something most with doctorates truly don't. Correlation, causation, confounding factors, understanding the motivators for finding "something", dodging oversimplifications, and understanding the slow timeline for real knowledge are all something the general public doesn't comprehend.

I like Scott Adams statement (I donk't recall the quote): Don't trust anyone who is absolutely sure about a complex issue. I guess that means the word "All" in your title is flawed!

Jess said...

There was a great article a while back in the NYer as well, that discusses the failure of many scientific studies to reproduce previously held "truths" that many other theorems were based upon for years afterward. It's really something how people seem to think that science is "truth" when actually it is the process of discovery. A lot of ego is involved in this faulty construction of reality so many hold dear.

Anonymous said...

Wait, I need to stop eating toast?