Persuasion and Science - Too Good To Be False
7th December 2007
The most recent issue of the Journal of the American Medical Association reports a study that is shocking and I’ll bet you dollars to donuts that it won’t get much attention. Just read the abstract:
Persistence of Contradicted Claims in the Literature
Athina Tatsioni, MD; Nikolaos G. Bonitsis, MD; John P. A. Ioannidis, MD
JAMA. 2007;298(21):2517-2526.
Context Some research findings based on observational epidemiology are contradicted by randomized trials, but may nevertheless still be supported in some scientific circles.
Objectives To evaluate the change over time in the content of citations for 2 highly cited epidemiological studies that proposed major cardiovascular benefits associated with vitamin E in 1993; and to understand how these benefits continued being defended in the literature, despite strong contradicting evidence from large randomized clinical trials (RCTs). To examine the generalizability of these findings, we also examined the extent of persistence of supporting citations for the highly cited and contradicted protective effects of beta-carotene on cancer and of estrogen on Alzheimer disease.
Data Sources For vitamin E, we sampled articles published in 1997, 2001, and 2005 (before, early, and late after publication of refuting evidence) that referenced the highly cited epidemiological studies and separately sampled articles published in 2005 and referencing the major contradicting RCT (HOPE trial). We also sampled articles published in 2006 that referenced highly cited articles proposing benefits associated with beta-carotene for cancer (published in 1981 and contradicted long ago by RCTs in 1994-1996) and estrogen for Alzheimer disease (published in 1996 and contradicted recently by RCTs in 2004).
Data Extraction The stance of the citing articles was rated as favorable, equivocal, and unfavorable to the intervention. We also recorded the range of counterarguments raised to defend effectiveness against contradicting evidence.
Results For the 2 vitamin E epidemiological studies, even in 2005, 50% of citing articles remained favorable. A favorable stance was independently less likely in more recent articles, specifically in articles that also cited the HOPE trial (odds ratio for 2001, 0.05 [95% confidence interval, 0.01-0.19; P < .001] and the odds ratio for 2005, 0.06 [95% confidence interval, 0.02-0.24; P < .001], as compared with 1997), and in general/internal medicine vs specialty journals. Among articles citing the HOPE trial in 2005, 41.4% were unfavorable. In 2006, 62.5% of articles referencing the highly cited article that had proposed beta-carotene and 61.7% of those referencing the highly cited article on estrogen effectiveness were still favorable; 100% and 96%, respectively, of the citations appeared in specialty journals; and citations were significantly less favorable (P = .001 and P = .009, respectively) when the major contradicting trials were also mentioned. Counterarguments defending vitamin E or estrogen included diverse selection and information biases and genuine differences across studies in participants, interventions, cointerventions, and outcomes. Favorable citations to beta-carotene, long after evidence contradicted its effectiveness, did not consider the contradicting evidence.
Conclusion Claims from highly cited observational studies persist and continue to be supported in the medical literature despite strong contradictory evidence from randomized trials.
Author Affiliations: Department of Hygiene and Epidemiology, (Drs Tatsioni, Bonitsis, and Ioannidis) and the Department of Dermatology (Dr Bonitsis), University of Ioannina School of Medicine; and the Biomedical Research Institute, Foundation for Research and Technology-Hellas (Dr Ioannidis), Ioannina, Greece; Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts University School of Medicine, Boston, Massachusetts (Drs Tatsioni and Ioannidis).
Even if you are not trained as a statistician you get the drift from the article title. Doctors persist in believing disproven claims. Let’s drill down a bit here.
The researchers in this article draw a distinction between information generated from “epidemiological” studies versus information generated from “randomized controlled trials (RCT).” These researchers put forth the claim that RCTs are the gold standard and if epi studies conflict with the experiments, science goes with the experiments. (If you want all the pain and glory of learning about these things you can start with this chapter from Steve’s Primer on research methods. You’ll like it. Really. It won’t hurt a bit.)
To demonstrate their point, the researchers use the example of Vitamin E studies. The initial findings of a positive effect for using Vitamin E came from a couple of epi studies published in the early 1990s. (I remember reading the NEJM one and using it in, of all things, a large lecture introductory media effects class - the kids were bedazzled.) An epi study is a type of survey where you ask a lot of people (millions sometimes), “Do you take Vitamin E?” and then wait 10 years and ask all the same people, “Are you still alive?” Now, of course, you ask a lot more than the Vitamin E question and you also ask for other interesting outcomes besides mortality, but you get the point.
Experimenters, by contrast, take a slightly different approach. They will get a smaller group of people (sometimes less than one hundred), give everyone a pill, but some folks are randomly assigned a real Vitamin E pill while the others get sugar pill, a placebo. The researchers then watch everyone for awhile and see if anyone dies or gets sick or files a lawsuit.
Experimenters typically mistrust epi researchers while epi researchers tend to view themselves as the true Children of Science. Let’s just say that there is a tension between the RCP folks and those epi mavens. Oddly enough, I am an experimentalist and I can also play statistics like chords on a guitar, so guess where my biases lie. Back to the opera.
We have frightening evidence that medical researchers and physicians and nurses and all those people who went to Schools of Medicine or Public Health, keep on believing things that are untrue. And not just believing, they keep on citing them, teaching them, defending them.
There is likely to be a brief debate about this article, especially from the epi community. Please make sure I’m not on the jury for this one because I’ve already tangled with the epi folks. They require budgets slightly larger than the national debt, typically find piddling effect sizes expressed in those lame 1.1 relative risk ratios, and then they think they’ve discovered blue sky, white snow, and a free lunch . . . not to be harsh or judgmental or unfair or wicked or even accurate about it.
Now, the thoughtful among you might be wondering what all of this has to do with persuasion besides Steve whaling away on some group of people you’ve never met.
Here: if scientists get it wrong and persist in getting it wrong, what explains their beliefs and behavior?
Persuasion theory and research.
Not health theory and research.
This research study demonstrates a common finding in persuasion research on biased, central route processing. Smart, thoughtful people will try to make the data fit their hypothesis even when it is apparent that the data don’t fit.
This is another way of saying that scientists and physicians and their ilk are human after all and behave according to the same rules of psychology that all us mere mortals obey. We tend to believe we are smart, consistent, and justified and we tend to make data fit our key beliefs more often than we make our beliefs fit “scientific reality.”
As I mentioned at the top of this post, it’s unlikely that this JAMA story will get much attention despite its critical importance. Here’s a quick illustration.
On the same day that JAMA published this research article, the New England Journal of Medicine published a study on childhood obesity and adult heart disease. Using the Googe news aggregator, I found that in the two days following these publications the childhood obesity study has been picked up in 350 different web sources, the majority of which are standard news sites like the New York Times or the Washington Post. By contrast, in the two days since the publication of this scientific bias study, the Google news aggregator lists fewer than a dozen sites carrying a story about it.
What’s interesting to me about this comparison is that the childhood obesity study is an epi-style method that finds about the same effect size as the Vitamin E studies. As we’ve seen with those epi studies on Vitamin E, we need to be extremely careful in accepting the conclusions until we get stronger evidence from experimental studies (RCTs).