If The Gloves Don’t Fit You Must Acquit
21st November 2011
Please contrast these two quotations.
Results demonstrated benefits of tailoring over both generic and control conditions and uncovered the degree of tailoring that produced the largest effects.
That from the Abstract. Now this from Results.
Hypotheses about the effects of tailoring were directional. We expected that tailored materials would be superior to generic and control conditions in prompting use of vegetables. If generic materials had any effect at all, they should be superior to the control group. Results of the t-tests of the residual days-and-ways scores were only partly confirming. The two tailored sites averaged +.21 above the overall mean residual, compared to -.42 for control pantries (t = 3.06, 478 df, p = .001, one-tailed). But days-and-ways scores of generic pantries were just as high, +.22 (t = 3.09, 460 df, p = .001, one-tailed). Overall, tailoring appeared to confer no advantages over standardized presentations.
Clarke, Evans, and Hovy present their findings from an interesting tailoring study in the September 2011 issue of Health Communication. It’s a field experiment testing how to encourage greater consumption of vegetables for patrons of inner city food pantries. And, it also talks out of both sides of the tailoring mouth claiming both success and failure. That first quote from the Abstract, which is what most readers will consider, comes across as if message tailoring works. Yet, that second, longer quote reports no difference between tailoring and a generic message. Of course, most readers of the paper probably don’t read that key paragraph; hey, the editors and the reviewers guarantee Methods and Results so all you need to read is Rationale and Discussion, right?
You cannot Abstract this research with a line about “demonstrated benefits of tailoring” over generic messages and then report no differences between the two conditions in Results. I appreciate that the team included other constructs in this study and found correlational support for them. That’s nice. But, the data show no difference, yet again, between a tailoring process and the mere generic message in a quasi-experimental test. More troubling still is the overall message effect, whether for tailored or generic messages, is a Smallish Windowpane. It’s roughly the size from Leslie Snyder’s communication intervention meta.
We’re now 20 years out from that meta and the field is still doing the same old same old. No one appears to be looking at the past and learning anything from it. As I’ve noted in this Blog, people keep doing Framing or Health Beliefs or Tailoring the same old way and obtaining the same old results. Sometimes folks will put a wrinkle on the Old Old Thing and obtain slightly better results, but everyone seems to be adding their own new wrinkles and instead of ending up with a theory and one sharp new crease, you have a theory that looks like a crushed ball of plastic wrap with wrinkles everywhere.
This paper reads in a smooth and persuasive style throughout that simply ignores the key finding of no difference. It is a shining example of All Bad Science Is Persuasive with a dancing narrative that reads more like a first draft from a Malcolm Gladwell effort. Sure, we have that untidy no difference, but if you know how to read the tea leaves, there’s a cupful of . . . what? The paper lyrically sings success in the foreground while grunting failure in the background as if the results are just a bad drummer hidden behind the stylings of the lead singer.
Look, this project was a serious piece of very hard work. The authors suffered through the Fed grant review process at USDA and all that jazz. They then executed a difficult, laborious, and time-intensive field study in tough circumstances. I rejoice in their diligence, patience, and fortitude.
But, the gloves don’t fit and we must acquit. Tailoring is not guilty of anything.