If you’d like, you can take the time to read like a good grad student this news article about the effect of multivitamins on cancer. Note all the details about rates, types, and impact. Or just read the lede for the general perspective.
After a series of conflicting reports about whether vitamin pills can stave off chronic disease, researchers announced on Wednesday that a large clinical trial of nearly 15,000 older male physicians followed for more than a decade found that those taking a daily multivitamin experienced 8 percent fewer cancers than the subjects taking dummy pills.
The rest of the article does a surprisingly good job of explaining a randomized controlled trial where some older male physicians got the multivitamin every day and others got a placebo. It includes a bunch of temperate quotes from experts that note the small effect within the experimental design and still see all the benefits. Except this omission, taken directly from the study abstract.
There was no significant difference in the risk of cancer mortality (multivitamin and placebo groups, 4.9 and 5.6 events, respectively, per 1000 person-years; HR, 0.88; 95% CI, 0.77-1.01; P = .07).
So. Among older professional men taking a pill that is sent to them every day (active or placebo), the rate of cancer expressed as a rate for person-years is decreased 8%, but the death rate from cancer is unchanged. Doesn’t that seem a bit odd? How can you have fewer cancers, but the same number of deaths? Here’s the trick.
If you read the article you will not find a simple statement that tells you how many men got cancer and died from cancer in the two groups, multivitamin or placebo, but only those events per 1000 person-years. Instead you have to dig around in the various tables and charts to find the raw numbers. Table 2 provides the details (click to enlarge).
Look on the top row for the Total Cancers. At the end of the row you see the Hazard Ratio, its confidence interval, and the statistical significance. Let’s look more carefully. This is an expensive, NIH funded study that got its money in part because it ran a randomized controlled trial, that random assignment of male physicians to drug condition, multivitamin or placebo. Experiments like this are very expensive in part because of the randomization which is often described in the medical literature as the gold standard (as opposed to the Golden Standard of Tooth Fairy Tales). The Fed paid a big price to get randomization and that top line in Table 2 provides the crucial, expensive data. Here’s a simple cross tabulation of the data.
|Number of Men
The experimental data here require a simple chi-square test. Randomization, especially with a sample this large, provides all the “adjustment” for confounders; in fact it would be crazy stupid to subject randomized experimental data to “adjustment for confounders.” Randomization is why you paid your tax money to fund this project. Let’s run a 2 X 2 chi-square test on this cross tabulation. It comes out like this: Chi-square (1) = 2.439, p (two tailed) = .118, w effect size = .012.
Kids, that p value of .118 means the test shows not even a statistically significant difference even though 1290 is smaller than 1379. That difference is within the range you’d expect from mere sampling error which is what statistical significance tests. And look at the w effect size of .012. A Small w would be .10 so this is one tenth as small as a Small Windowpane! The raw incidence of cancer is not different between the two groups even with this very large sample size of 14,000 cases. Yet, the research and news reports all claim that taking multivitamins reduces cancer. Even that Table 2 shows a statistically significant p < .04 value. How is this possible?
First, run a General Semantics Persuasion Play. Carefully select Words and Numbers that seem to point to the Thing, but quite exactly do not. Use the Words, Randomized Controlled Trial. Then when the real Numbers, the ones we just ran a chi-square on, do not deliver the desired Thing, change the Numbers by adjusting them. As the researchers note in Table 2:
C Adjusted for age, PHS Cohort (original PHS I, new PHS II participant), and randomized treatment, and stratified on baseline cancer.
Look. If you are doing a randomized controlled experiment and then you adjust the outcomes, you’ve just broken the randomization and converted the experiment into an observational Tooth Fairy Tale. You no longer have a randomized controlled trial. Yet the researchers continue to use the old Words of “randomized controlled trial” with new Numbers from an observational analysis. That’s why when you look at Table 2 they will report the raw numbers as producing a statistically significant outcome at p < .04. (And look at the confidence intervals for the Hazard Ratio out to three decimal places at .998 which is actually 1.00 which is non significant, but when you provide the false precision of three decimal places it looks like you are doing science when you’re just doing persuasion.)
As we’ve seen, the simple chi-square is not significant and worse still even if you believe that this General Semantics Persuasion Play is according to Hoyle you still have that w effect score of .01, one tenth of the Small w at .10. Thus, say you are doing an experiment but report observational numbers; say you report the adjusted statistical significance but ignore the effect size. And, it is that persuasion trick of hiding in plain sight. They say exactly what they did, just not in a way that exactly points to what they really did.
Second, they artfully reframe the outcome as events per 1000 person-years instead of plain old incidence. In other words, they add time to diagnosis to the incidence of cancer. If you instead reframe the outcome as person-years, then you do get slightly better news. This is a tricky way to put it.
Multivitamins do not reduce the number of cancer cases, they delay the onset of cancer in a very small number of men by a few weeks or months. Hey, take this pill and . . . what? You’ve got the same likelihood of dying from cancer regardless of whether you take an active ingredient or a placebo. You have the same likelihood of getting cancer. It just takes a bit longer to get the cancer diagnosis. Maybe. Analysis of the experiment conducted the way God and Ronald Fisher intended it shows null results. Only when you employ Grimm Brothers story telling do you get a Fairy Tale.
Yet, that is clearly not the thrust of the research or the NYTimes article. Both make it sound as if taking a daily multivitamin protects against cancer when that is not exactly true. Everyone still got cancer and died from it regardless of vitamin or placebo. That is exactly what the results say and that is not a particularly tricky, nuanced, subtle, or complicated headline. Yet, the researchers claim:
In this large-scale, randomized, placebo-controlled trial among middle-aged and older men, long-term daily multivitamin use had a modest but statistically significant reduction in the primary end point of total cancer after more than a decade of treatment and follow-up.
This is not a correct statement. The results they report are not from a randomized, placebo-controlled trial, but are from a broken experiment that is converted into an observational design with all the adjustments. The results from the experiment show that there is no difference between the two conditions even with the large sample size and all that statistical power. If, however, you break the randomization, adjust scores for confounders, and express the outcome for length of time along with the incidence of a cancer diagnosis, then you find an extremely Small effect that is also statistically significant.
But, it’s in JAMA and making headlines in major media sources. Consider the practical payoffs from this persuasion play.
Go long on GNC stock. This is nothing but great free news for people selling stuff that gives you no benefit except for excreting more expensive urine. Do the math on Take This Pill. Let’s say that we’ve got 180 million adults in the US. They take a pill every day. Go online and find the cost of your favorite pill. I saw ranges from about 50 cents to a dollar, so let’s model the range. Take This Pill would cost between $32 to $65 billion each year. And what does that buy? If you believe in Fairy Tales – a slightly longer delay in cancer diagnosis, no decrease in number of cancers or number of cancer deaths. If you believe in science – nothing.
Furthermore, all those multivitamin guys are now bullet proof against any government interference when they can now claim in their advertising that their multivitamins have been proven by the National Institutes of Health to reduce to risk of cancer! Good grief, you couldn’t make this up. You can hear the screams coming from the regulators at the Food and Drug Administration who now have to approve ads from Suzanne Somers extolling the benefits of a natural product that includes multivitamins.
Also, go short on sugary drink regulations because manufacturers will now add multivitamins to the drink and evade the obesity argument with the cancer prevention argument. Hey, which is scarier, getting fat or getting cancer? Merely adding a slug of beta carotene to any food or beverage nicely complicates any regulatory scheme.
You now see the painful side effects of people calling themselves scientists who bend the standards of judgment to increase their odds of publication (and tenure and promotion and grants and awards and on and on). Sure, you just got a hit in a high prestige peer review journal. And you also sold billions of dollars of product that you know will make no practical difference in mortality.
Oh. There’s another practical side effect to the Take This Pill Persuasion Play. It appears to justify the hundreds of thousands of dollars of taxpayer money that funded this research. NIH is in a very uncomfortable position here of dropping a large dime on this research that had to run the General Semantics persuasion play to evade, confuse, and distract thoughtful review. All the players on this research can think that they publicly dodged a bullet by getting millions in grants that made something out of nothing, not with science, but with persuasion. If I was the program officer that funded this project, I’d be very nervous that some reactionary Republican Senator might look at the dollar amount of funding on this, see the sophistical statistics that make the weaker argument seem the stronger, and start asking questions in public.
You want to know how to do sophistical statistics? Take This Pill.
Gaziano J, Sesso HD, Christen WG, et al. Multivitamins in the Prevention of Cancer in Men: The Physicians’ Health Study II Randomized Controlled Trial. JAMA. 2012;():1-10.
P.S. In an earlier PB Post (Tooth Fairy Tales with Omega-3) I joked about Tooth Fairies making confounder adjustments with “randomization.” Apparently the Harvard guys didn’t get the joke!
P.P.S. In yet another prior PB post (Pop Goes The Weasel) we saw the Tooth Fairy humor looking at physicians drinking soda pop and observed the same sophistical statistics there we see here. Yeah. The soda pop study and this multivitamin study are on the same group of people. Here’s the fun part. In the Pop Goes the Weasel, the Tooth Fairies had to adjust the data to get their statistical significance.
. . . after adjusting for age, smoking, physical activity, alcohol, multivitamins, family history, diet quality, energy intake, BMI, pre-enrollment weight change and dieting.
Thus, the Harvard guys prove that 0 + 0 = p <.05!