The Perils of Small Effects Science for Persuasion

Today the world learns that calorie restricted diets help monkeys live longer lives.  You can see a current lineup of news sources and how they are covering this story and see that folks generally view this as important news for humans living longer and healthier lives.  You can certainly find cautionary quotes in some articles about replication, extension to humans, practical barriers, still-a-long-way-to-a-pill notations, but generally, it’s all good.

Baboon Eating CarrotMonkeys Eating Popsicles

Except it isn’t.  This is lousy news and makes things worse for most people while making it better for people who write grants.  To borrow from Denise Richards:  It’s complicated.  But understandable.

1.  This new report published in Science shows another example of how seriously restricting an organism’s daily calorie intake (about 30% less than normal), extends life and improves health.  The first demonstration of this effect occurred in 1935 and has since been studied in bugs, yeasts, rats, and monkeys.  The effect holds across these diverse species.  At a detailed level researchers find exquisite changes in DNA machinery in these restricted calories critters, meaning that a large behavior like diet can affect the most delicate and tiny element of biological functioning.  So, we have an effect at a large level of analysis (eat less, live longer), across many species, and a plausible biological mechanism, plus much of this work is experimental, meaning we’ve got great control over the variables.  Hubba-hubba.

2.  Consider the specific outcomes from this latest monkey study.  The University of Wisconsin researchers randomly assigned 38 adult monkeys to a “free range” diet, meaning the monkeys ate as much as they wanted while another 38 monkeys got the “restricted calorie” diet.  This was done in a monkey lab so these critters were not running across the savannah, but living in a carefully controlled setting.  We know what they ate.

The researchers tracked these 76 monkeys for over 10 years and that’s a long time in monkey years (average life span is 27 years).  Let’s look at a windowpane illustration of the outcome after 10 years.  We have two groups (treatment versus control) and two outcomes (alive or dead).  It looks like this.

Alive

Dead

Total

Treatment

33

05

38

Control

24

14

38

 

If you do an appropriate test of statistical significance, in this case a chi-square, you derive a value of 5.68, with a p of less than .017, and a w effect size of .27.  Since these monkeys were apparently randomized to condition (the restricted calorie treatment or control), tests of statistical significance are warranted.  You’ll see a lot of quantitative voodoo with epi observational studies where there is no randomization of anything except the sequence of items which is the Mark of the Beast by my training, so we’re on good scientific footing here because this is a true experiment.

However, while we can remove sampling error as a rival explanation for the outcome differences, consider that outcome.  Nearly 3 times (14/5) as many monkeys in the control condition died compared to the treatment condition.  That sounds like a lot, but please note the effect size estimate, the w, of .27.  That’s a Small Windowpane effect, roughly equivalent to a d of .20 or an r of .10.  Here’s how “small” it is.  If instead of 5 treatment monkeys dying, it had been 7 monkeys, the test of statistical significance would have failed to reach the conventional .05 probability level.  And that would have still been a 2 to 1 ratio (14/7).

This outcome looks marvelous when you start playing the guitar with ratios, but when you get down to the details, the effect just barely exceeds random variation.  And this is with a highly controlled experimental study where we know exactly what each monkey is getting and we can measure with great precision all the variables we study.  And, consider, too, that this is a 30% reduction in normal calorie intake every day of life for every monkey in the treatment group.  That is a huge change in normal behavior, just to obtain an effect that can be made to look large or beneficial or important, but that is actually just barely detectable over random variation.

3.  Consider, now, my problem as a persuasion campaign guy trying to mount an intervention aimed at influencing people to eat 30% fewer calories every day for the rest of their lives.  And, this 30% reduction is not aimed at their current consumption level which is way too high to begin with.  It’s aimed at the “normal” or “average” level which is about 2,200 calories for a normal weight adult.  Thus, I’m gonna tell people who are eating 3,000 calories or more a day (and are overweight or obese as a result), that they need to get down to 1,200 calories a day.  You’ll live LONGER because the monkey science says so.  It’s experimental.  It’s replicated.  It’s cross-species.  It’s diddles your DNA.  It’s Science!

PT BarnumAnd, for what real effect?  A possible gain in life span.  By one reasonable estimate (PDF), the standard deviation for life expectancy is about 15 years.  With a small of effect of .2 we are talking about an average expected gain in life of 3 years.  Thus, if you cut down to 1,800 calories a day for your entire life, you can expect to live on average 3 more years.

I’ll need to bring back P.T. Barnum from the dead to sell that proposition.

4.  Hey, instead of people having to do the heavy lifting of big calorie restriction, maybe we can make a pill that creates the same effect.  Now, that’s a persuasion proposition of an entirely different color.  Instead of poor old me having to design and run an intervention that tries to convince people to eat 1,200 calories a day, all I have to do is find the pill, get a patent that kills any potential competitor, unleash a media ad campaign, and rake in the big bucks.  But, remember, to achieve this gain, you have to take this pill your entire life.  That might cost some money.  And, of course, there’s no real serious chance of side effects like an erection lasting longer than four hours, vaginal dryness, wretching, runs, liver damage, or birth defects since you’re taking a pill that will affect the molecular functioning of your DNA!?!

5.  There is great science in this line of research.  Serious people are working seriously and have reasonable results to pursue.  For science.  Not for real life.  The newspaper buzz on this science is, to put it mildly, demented, irrational, and absurd.  And, the scientific community is playing along with the delusion, too.  The summary story in Science that describes this report puts lipstick on the pig.  Nowhere does anyone raise any of the points a lightweight persuasion “social scientist” like me makes.  And, the points I make are valid.

This is good science.

It is bad for persuasion.

It is bad for the general population.

It is good for journalists and scientists.

It is normal life.

P.S. The pictures are inflammatory.  The primate eating the carrot is not a monkey, but a baboon.  The monkeys eating the popsicles are clearly not experimental participants.  And, calorie control is not achieved by the type of food (carrots versus popsicles) but rather by the total number of calories in the yummy monkey chow.  And, that’s a cartoon image of P.T. Barnum, not a photo, but it is accurate.

P.P.S.  Don’t get confused here.  This is good science, but the reporting and PR spin on it is Very Bad.  I strongly support the science and would recommend funding for this line of research if I was on a grant committee or the Branch Chief evaluating my own scientists’ proposals.  I don’t want the results bandied about in the New York Times as if journalists know the difference between baboons and monkeys or between telomeres and mallomars.  They don’t, but they offer the prestige of public discussion and some scientists like that more than peer review.  Cassandra has spoken!