On August 30, 2010, Back, Kufner, and Egloff published a short study with the peer review journal, Psychological Science. They got a Really Big database of over 400,000 text messages sent on September 11, 2001. They fed those messages into a well established software program, the Linguistic Inquiry and Word Count (LIWC), for analyzing the semantic content of this database to measure the emotions being expressed that day. They found a correlation of .84 between angry text messages and time indicating that people got angrier, a lot angrier, across the day. From this analysis of a Really Big dataset, the researchers concluded:

In sum, we investigated a large data set providing unobtrusive behavioral measures of negative emotions actually expressed during September 11. We were able to determine that people did not react primarily with sadness; that they experienced a number of anxiety outbursts, but recovered quickly; and that they steadily became angrier . . . anger is known to predict moral outrage and a desire for vengeance, which — once aroused — seem to require an outlet (Skitka et al., 2004). This might help to explain individual acts of discrimination following the attacks, as well as societal responses such as political intolerance and confrontational policy.

I would argue that their conclusions do not follow from this research, especially given they employed Observational Research where Really Big can seem Really Important, but never forget it’s Really Convenient and Biased to boot. And, how you go from Angry Text Messages to Intolerance and Confrontation seems to require proof of intervening stages, but whether it is calorie counts on menus, cell phones and driving, or statistically insignificant variation in something called Global Temperature, some folks have no trouble moving from a correlation to causality and thence to law and regulation. But, people smarter than I found a different reason to dispute this study.

See, on May 9, 2011, less than a year later, Cynthia Pury published her observations in same journal about the Angry American research from Back et al. Pury noted a slight problem with one case in this Really Big dataset.

The data contained many technical codes; thus, Back et al. counted only words recognized by LIWC. However, this procedure did not exclude automatically generated messages. Consequently, LIWC words in such messages were counted, even if the words lacked emotional meaning in context. Furthermore, computers can send messages with superhuman frequency, turning an otherwise minor measurement error into a serious confound. This confound can be detected by treating individual text messages as primary units, reading samples of each key word in context, and looking for repeating false positives.

Stated another way, Back et al. did not inspect that database of 400,000 messages and just assumed every text message in the database was sent by a human to another human. It never occurred to them that the database might include technical messages sent between devices (i.e. the pager and its server). Pury found that over one third of the messages in the database were indeed these technical exchanges and more importantly, every one contained the word, “critical,” in the message. Worse still, all of these technical messages came from just one device, a single pager.

Now it turns out that the well established software program that determines emotion from semantic content counts the word, “critical,” as an anger word. More of these “critical” messages were sent between the pager and server as the day progressed probably because of network failure from heavy volume. Thus, Back et al. determined on September 11, Americans got angrier, more intolerant, and more discriminatory simply without carefully inspecting the dataset and its contents.  When the data from this one pager – which only carry a technical message between the device and its server – are removed from the dataset, the Angry, Intolerant, and Confrontational American Effect disappears.

Back and colleagues responded in Psychological Science on May 13, 2011, the week after Pury’s note. They confirmed Pury’s analysis of their error. They provided a revised conclusion,

As Pury’s (2011) analyses suggest, however, the timeline of anger was not as straightforward as indicated in our original analyses . . . Additional analyses and sources of data will be needed for a thorough evaluation of the course of anger on September 11, 2001.

You could put it that way. Or maybe . . .

This is about as embarrassing as it gets in research publication. I’m sure that everyone who’s ever published peer review science experienced a near heart attack merely reading this exchange. It’s your worst nightmare. You bust your butt doing a study, turn everything inside out and upside down three times, show your results to your worst enemy, and when everything seems copasetic you send it in for another beating at peer review. It passes, you publish, then one day you open your emailer program and find a message with ?????? in the subject line from someone you don’t know from Adam, Eve, or the Serpent. You read a polite inquiry about inconsistencies and technical questions about methods and you’re thinking, “Who is this knucklehead?”

But, because you’re doing science, you take this seriously and run a quick check of the database looking for a particular pager and the key term, Critical, and then you feel your heart constrict even before your mind can explain why. You run a sort routine on the data and, BOOM, your screen fills up with a gazillion cases of CRITICAL from the same pager.

Your life flashes before your eyes. A moment from a Methods 101 seminar. A tenure committee. A crying grad student. An empty office. Then you contact your coauthors who repeat your movie, but for themselves. The gift keeps on giving as you verify the biggest mistake of your life, there in black and white, forever in digital storage, like a brain in a bottle filled with formaldehyde, except it’s your reputation in a peer review journal that will endure until the last scientist and the last storage drive.

And, too, something like this, but at a lower intensity, is going on with the journal editor and reviewers. Sure, you can blame a technical error like this on the authors; you can’t and shouldn’t analyze their data for them. But, nobody in charge caught this whopper. Due diligence? Let’s guess how the attributional search on this one will work out.

This is the bad news about peer review research. People make huge errors. But, the good news in this is that Really Big errors tend to get caught and the literature becomes self-correcting. The additional good news is that errors like this rarely seem to occur. Even those Errata notes (where authors catch their own errors) are relatively rare and most exchanges between teams of authors typically turn more on He Said, I Did Not, You Did, Too! rather than fabulous errors like this. Thus, peer review not only seems to publish fewer whoppers, it can also catch them and hold them up to the light of day.

In retrospect all of the errors from all the players in this may seem more apparent now than anyone could have realized then.  I’m not so sure. My radar does go off when I read an observational methodology like this that reports a correlation of .84 between a psychological variable and anything living or dead, real or imagined. With this kind of data and all the measurement and sampling error in it, a correlation of this size is a virtual identity, like parallel forms of a proven self report scale. Nobody – the authors, the reviewers, the editor – nobody flagged on this?

And then what was the source of the database, those 400,000 text messages? What, AT&T? Sprint? Some public record from a 9-11 Commission Report? Nope. Hold on to your BVDs or your panties or both.

WikiLeaks.

Yeah. The researchers went online to WikiLeaks and simply downloaded this double-secret, Dick Cheney/George Bush/CIA database. That’s why it was so easy for Cynthia Pury to catch the stupid error Back, Kufner, and Egloff committed. She, too, simply went to the WikiLeaks link, downloaded the file, and opened it and this can of worms. There is no provenance on this database, how it was built, how it was acquired, who accessed it, whether it was manipulated for any purpose. It’s just a hodgepodge of messages and technical codes that the scientists and freedom fighters at WikiLeaks acquired and posted online to expose the wrongdoings of governments. And no one thought that peculiar from a scientific point of view. Of course, Americans and America jumped on the Hate Wagon after 9-11. Gitmo! Waterboarding! Infringed Civil Liberties! The Angry Left! Oops. That last one doesn’t belong there.

Imagine instead if this had been from a Fox News server. Don’t change anything else, just that provenance. Think how the researchers, reviewers, and editor would have responded.

While I see a clear Biased Process in Back et al. conclusions about Angry Americans and all that zealotry, Back, Kufner, and Egloff are made to look foolish not for their silly, fevered, and unwarranted conclusions, but because they don’t know how to inspect a database. And, the journal editor and reviewers look incompetent because of their political beliefs. Who needs to think about a correlation of .84 or a wildly biased database source like WikiLeaks when we’ve got the right conclusion about anger, hate, and intolerance in America and Americans.

All Bad Science Is Persuasive, baby.

Mitja D. Back, Albrecht C.P. Kufner, and Boris Egloff
The Emotional Timeline of September 11, 2001 Psychological Science October 2010 21: 1417-1419, first published on August 30, 2010, doi:10.1177/0956797610382124

Cynthia L.S. Pury
Automation Can Lead to Confounds in Text Analysis: Back, Küfner, and Egloff (2010) and the Not-So-Angry Americans Psychological Science May 2011, first published on May 9, 2011, doi:10.1177/0956797611408735

Mitja D. Back, Albrecht C.P. Kufner, and Boris Egloff
“Automatic or the People?”: Anger on September 11, 2001, and Lessons Learned for the Analysis of Large Digital Data Sets Psychological Science May 2011, first published on May 13, 2011,
doi:10.1177/0956797611409592

Dissonance proves human nature. People love that for which they suffer as Leon Festinger observed and the suffering starts with inconsistencies. You love your child who then pukes all over your laptop computer. When these inconsistencies arise, dissonance inflates, producing tension. And, as the theory predicts, often you resolve that tension by loving your puking child more. You resolve the suffering by valuing that which provokes the suffering.

But, how does this actually occur in people?

Consider this interesting fMRI study (pdf) that explored how the brain reacts during dissonance arousal and reduction. We’ll put people in the brain scanner, an experience most people find uncomfortable and claustrophobic. It’s not fun. We give them attitude statements (Target) that ask for their reaction in the scanner along with other attitude statements (Neutral) not related to the scanner. Start with the Control Group.

Participants in the control group were told to respond to the target sentences as though they were enjoying the scanner and the task, regardless of whether they were actually enjoying the experience. Furthermore, they were informed that they would receive an additional dollar for each sentence that they responded to in this way. They were instructed to respond honestly to the other (neutral) sentences.

And, now for the Treatment Group.

Participants in the dissonance group were also instructed on how to respond to the stimuli. They were then told that a patient had been scheduled to be scanned after them and was to perform a similar task in the scanner. This patient, the participants were told, was now in the scanner control room, watching the screen of the experimental control computer, and was very nervous and uncomfortable about the upcoming scanning session. The participants were then told that several of the sentences were about their attitudes toward the scanner and the task and were asked if they would be willing to respond as though they were enjoying being in the scanner and performing the task, regardless of how they actually felt about the experience. This, they were told, might put the patient’s mind at ease, as the patient in the control room could see the responses on screen. We reasoned that this would be analogous to making a counter-attitudinal argument.

This is a sweet little manipulation for a scanner dissonance experiment. Rather than try to replicate past dissonance studies, this experiment makes the experiment itself part of the dissonance manipulation. People are asked to fake it for another person – in other words, performing a counter-attitudinal behavior (I’m acting like I enjoy this unpleasant experience). The control group also pretends, but in a direct, obvious, and compensated fashion. Past research has strongly demonstrated that this manipulation creates an external attribution (Why am I suffering? They paid me to act!).

The researchers also included a technical manipulation that was handled through a Solomon Four Group Design, which, it turns out, didn’t matter that much, so I’ll just note that and glide on.

The two key outcomes are how the two groups, Dissonance and Control, 1) reacted to those attitude statements, both Target and Neutral, and 2) showed any variation in brain activity. Consider the attitude statements.

Analysis of composite final attitude scores by means of an experimental group (dissonance, control) by pretest presence (present, not present) ANOVA verified a main effect of experimental group (F1,39 = 12.36, P = 0.001). This showed that scanner enjoyment was greater for the without-pretest dissonance group (M = 6.3, s.d. = 0.8) and withpretest dissonance group (M = 6.0, s.d. = 1.1) than for the withoutpretest control group (M = 5.1, s.d. = 1.3) and the with-pretest control group (M = 4.5, s.d. = 1.3). Individual t tests verified this effect for both the without-pretests groups (t22 = 2.51, P = 0.020) and the with-pretest groups (t17 = 2.43, P = 0.026). These results verified the basic cognitive dissonance finding.

Okay, I know some of you struggled through that and I appreciate the effort. Quite simply, the attitude difference between the Dissonance and Control groups was quite Large in Windowpane terms. That t(22) = 2.51, translates into a d over 1.00, roughly a 20/80 Windowpane. Dissonance really messes with your attitudes. Now, what about brain activity?

The research team measured brain activity in a variety of different areas, but expected the Dissonance expect only in specific locations. As the authors put it:

One candidate region for the detection and processing of cognitive dissonance is the dorsal anterior cingulate cortex (dACC). We and others have proposed that one of the dACC’s functions in cognition is to detect conflicts between active, but incompatible, streams of information processing13–15, such as between the color and the meaning of a word in the Stroop task16,17. dACC activation is consistently related to the amount of conflict occurring in such tasks.

And, indeed this is what they find. They offer a variety of statistical tests and figures to demonstrate the outcome. I’ll just share one.

These correlation plots display the relationship between activation and attitude for the key brain areas. The visual display in this case actually does a poor job of showing just how large the difference between the Dissonance and Control groups is. Note that the correlations in the Control group are around r = .10 to .30 and often in the wrong direction while for the Dissonance group they are all around r = .60 and in the right direction. These are Huge Windowpane effects.

This is pretty good evidence that the experiment did produce dissonance arousal and reduction and we’ve got good evidence that links dissonance to brain activation in areas related to conflicted information processing.  I’ll quote the authors.

These findings are consistent with a number of prior observations. Both cognitive dissonance and dACC and anterior insula activation have been associated with negative affect and autonomic arousal. These regions might therefore be responsible for representing or triggering the negative affect and related autonomic arousal associated with the dissonance . . . In short, our results are consistent with theories of cognitive dissonance that emphasize conflict between different cognitions, such as the original theory.

Realize the quality of the science here that again replicates key elements of Dissonance Theory. We’ve got a well done experiment with randomization, comparison, control, and counting. It connects a growing body of evidence in the neurosciences with behavioral studies going back to the 1950s. In other words, Dissonance is not some brand name magic sold by the gypsy boys and girls trying to make a buck. It is a well documented fact of human nature that describes, predicts, and explains how people change.

Dissonance is a play for genuine persuasion masters who truly know how to make change. Inconsistencies make people vulnerable to persuasion plays. They provoke intense, fast, and deep reactions both in physiology and psychology. Most of the time, people in the throes of Dissonance have no awareness of the effect. They just have an inconsistency problem to solve or so they think.

van Veen V, Krug MK, Schooler JW, and Carter CS. (2009). Neural activity predicts attitude change in cognitive dissonance. Nature Neuroscience, 12(11), 1469-74. Epub 2009 Sep 16.

I agree that we owe everything to Shakespeare, even if to say so invites scorn from the usual rabblement:  comma counters, “cultural” materialists, new and newer historicists, gender commissars, and all the other academic imposters, mock journalists, inchoate rhapsodes, and good spellers.

Anatomy of Influence, p. 129 by Harold Bloom.

Comma counters!

Good spellers!

Midway upon the journey of our life
I found myself within a forest dark,
For the straightforward pathway had been lost.
Dante, Inferno, Canto 1, Longfellow translation

I’ve been an irregular member of the Society of Personality and Social Psychology and a regular reader of the great journal, Personality and Social Psychology Bulletin. I received recently a membership email describing various changes and improvements to the association, including:

For many years the society has studied the merits of hiring a public information officer to help raise awareness of research by social and personality psychologists. After much deliberation the Executive Committee approved the creation of this position and a search committee is currently seeking candidates to fill this important role for the society. Given the success of books written by the likes of Malcolm Gladwell, it is clear that the public is eager to know more about our research. Our new public information officer will help make that happen.

A public information officer? Is that what we’re calling the Ministry of Propaganda nowadays? And, Malcolm Gladwell? An eager public awaits. Where’s the Facebook page so I can like you? Yeah, let’s tweet our fMRIs!

I appreciate the complexities of the contemporary research and media mindscape. But, be careful what you aspire to. Gatsby remains the cautionary consideration.

Or, for those so inclined, remember:

No one can serve two masters; for either he will hate the one and love the other, or else he will be loyal to the one and despise the other. You cannot serve God and mammon. Matthew 6:24

And, to go Old Testament:

And Elijah came to all the people, and said, “How long will you falter between two opinions? If the LORD is God, follow Him; but if Baal, follow him.” 1 Kings 18:21

But, to stay on Blog and Primer . . .

All Bad Science Is Persuasive.

The latest persuasion news in brief from the third issue of 2011 from the journal, Health Psychology. Things are going to start happening now!

Jane, You Ignorant Slut, and Null Results

Michael et al. analyzed results from the Women’s Health Initiative study on breast cancer and its relationship with stress. They published their findings in HP 2009. In the new issue of HP, Coyne and Johansen take exception to the findings Michael et al. report. And, then in the same issue, Michael et al. respond, completing the rhetorical trifecta of point, counterpoint, countercounterpoint so delightfully scorched in this SNL skit with Dan Ackroyd and Jane Curtin.

The controversy turns on a pile of null results from various raw and adjusted correlations between cancer incidence and self reported stress (and social support). Michael et al. note all the null and conclude,

Overall in this prospective cohort study, we did not observe a significant association of breast cancer with life events and we found limited evidence of an interaction between life events and social support.

Here’s the Table they supplied on the “limited evidence” for the interaction effect. Think about it.

Coyne and Johansen find this description inaccurate and argue,

Using data from the Women’s Health Initiative, Michael and colleagues (2009) suggested an association between stress and the incidence of breast cancer. However, we believe their results and those from other studies failed to confirm that stress is a risk factor of breast cancer.

Michael et al. respond,

In conclusion, we regret that Coyne and Johansen misrepresent our work. Additionally, we remind readers that while our study provides no support for an independent association between stressful life events and breast cancer risk, a complete causal mechanism for cancer may well include complex interactions of psychosocial factors.

Having played all the parts in this skit a couple of times, both in print and in conference palaver, I must confess complete frustration. As an actor in the play, I learned little about myself, the Other Guys, or the issue. And from watching several of these Albee-esque exchanges at various distances, I confess more frustration though leavened with entertainment from the drama.

I think it all boils down to that Table. If your next grant application is on stress and breast cancer, you might see a glimmer of hope for Future Research. If you are a stone cold scientist who cares not from where the next meal is coming, you laugh like Howard Roark in The Fountainhead.

Hey, I don’t just hate null results; I hate small effects in observational designs.  I get Coyne and Johansen.   I nonetheless think that Michael et al. played all this pretty well.  Stress research is slowly getting more sophisticated and virtually everyone has moved past the simple Stress Kills You!!! hypothesis, leaving just zealots who only embarrass themselves in public.  I could yell at Michael et al. for their assertion about “limited evidence” for the interaction.   Like Coyne and Johansen, I’d argue that the one hazard ratio of 1.20 in the Table within the pile of nullity is nothing more than randomness and certainly nothing any theory of stress and cancer I’ve read would predict.   And, don’t forget, WHI is still an observational design.  Anyone who professes any faith in Stress Plus Something Else Causes Breast Cancer is pretty much a snake handler with these data.

Not that there’s anything wrong with Snake Handling and faith.

Michael, Y. L., Carlson, N. E., Chlebowski, R. T., Aickin, M., Weihs, K. L., Ockene, J. K., . . . Ritenbaugh, C. (2009). Influence of stressors on breast cancer incidence in the Women’s Health Initiative. Health Psychology, 28(2), 137-146. doi:10.1037/a0012982

Coyne, J. C., & Johansen, C. (2011). Confirmatory bias and the persistent influence of discredited data in interpreting the stress-cancer link: Commentary on Michael et al. (2009). Health Psychology, 30(3), 374-375. doi:10.1037/a0022736

Michael, Y. L., Carlson, N. E., Bowen, D. J., & Ritenbaugh, C. (2011). Rebuttal re: “Confirmatory bias and the persistent influence of discredited data in interpreting the stress-cancer link: Commentary on Michael et al. (2009)”. Health Psychology, 30(3), 375-376. doi:10.1037/a0022737

Really Long Term II

Implementation Intentions is the killer app for practical persuasion. A quick glance through this Blog or a good science search engine will prove that. Here’s yet another application, this one reporting a two year follow up. Briefly the II team provided either control (usual treatment) or II planning for women seeking services related to birth control. As the authors note in the first paragraph of Discussion . . .

The present findings provide one of the longest objective follow ups in implementation intention research to date, and indicate that the effect of implementation intention formation on contraceptive outcomes is durable over 2 years. There were reliable differences in consultation outcome at a family planning clinic between control and implementation intention conditions over a 2-year period. In particular, rates of consultation for contraceptive supplies among implementation intention participants were 40% higher than the rate observed among control participants, whereas rates of emergency contraception, pregnancy testing, and positive pregnancy tests were 19%, 33%, and 43% lower, respectively. Findings also indicated that implementation intentions were especially effective in promoting positive outcomes among participants who were at greatest risk of pregnancy at the outset of the study.

How can you not use II?

Abstract

Objective: To assess the long-term impact of implementation intention formation in reducing consultations for emergency contraception and pregnancy testing among teenage women. Design: Teenage women visiting a family planning clinic were randomly assigned to implementation intention versus control conditions. Main outcome measures: Objective measures of consultation outcomes were obtained from clinic records at 2-year follow-up (N _ 227). Results: Rates of consultation for emergency contraception and pregnancy testing in the implementation intentions condition were 19% and 33% lower, respectively, compared to the rates observed in the control condition. Pregnancy rates were 43% lower. Intervention participants who consulted for emergency contraception and pregnancy testing at baseline were more than twice as likely to change to consulting for contraceptive supplies over the follow-up period compared to equivalent control participants (19% vs. 9%). Conclusion: The impact of implementation intention formation on reducing pregnancy risk among teenagers is durable over 2 years. Implementation intentions were successful in changing behavior among precisely those participants who were at greatest risk of becoming pregnant.

Martin, J., Sheeran, P., Slade, P., Wright, A., & Dibble, T. (2011). Durable effects of implementation intentions: Reduced rates of confirmed pregnancy at 2 years. Health Psychology, 30(3), 368-373.

doi:10.1037/a0022739