The public hype on Facebook as a persuasion money-making machine is an excellent case study of how vampires get Other Guys to believe that smoke and mirrors is persuasion. If you want to understand how to use persuasion to sell a company, study the history of Facebook. But, if you want to use Facebook for persuasion yourself on your Other Guys, the public evidence to date suggests a lot of sand and ice. Today we’ll look at what should be a sweet spot for Facebook advertising: Political campaigning.
Broockman and Green conducted (PDF) two randomized and controlled field studies of political advertising through Facebook for two actual elections held in 2012 for legislative races in California. Here are quick descriptions of the two campaigns.
In the first study, a candidate for state legislature deployed advertisements to randomly selected segments of his constituency. The collaborating candidate was a Republican running for state legislature. The candidate’s opponent was a longstanding Democratic incumbent who was running for re-election in a newly drawn district with a partisan composition that leaned Republican, giving the challenger a reasonable chance to win the seat.
For the second field experiment:
First, rather than collaborating with a relatively unknown candidate running for state legislature, we collaborated with a viable candidate running for Congress. This candidate enjoyed much higher name recognition prior to the launch of our study, and the contest itself was of much higher salience.
So, we’ve got a couple of different campaigns here. The first is with a relative unknown running against an incumbent, but in a highly competitive district while the second features a candidate with higher name recognition for larger office. And, we see that the ad campaigns vary the party so we’ve got one for a Republican and another for a Democrat. The two Locals are different.
You need to read the article for the details on the design and execution of the two ad campaigns. It’s complicated as practical persuasion always is and you will find it faster to read the paper than to wade through my technical description. I’ll put it this way: Any panther doing practical persuasion for themselves through Facebook would proceed like this. Stated another way, the researchers bought high Exposure with highly targeted Other Guy Voters and ran all the standard kinds of persuasion plays available through Facebook. This is exactly what Facebook is designed to deliver.
What about that Exposure? Did the purchase plan deliver messages on target?
During the course of the week-long advertising campaign, the Facebook ad interface reported that essentially every single person who could have seen the ads on Facebook did indeed see them (in Facebook parlance, the number of ‘targeted’ individuals was identical to the number of individuals ‘reached’)—5,012 users in the family treatment, 4,752 users in the character treatment, and 4,970 in the policy treatment, or 14,734 Facebook users in all. 19,377 voters on the voter file were assigned to these clusters. Facebook’s records suggest that over the course of the week the typical targeted person saw the ads about three dozen times.
Facebook claimed that the ad campaign nailed the targeted Other Guys with 36 messages for the candidate over 7 days. Facebook asserted that all of the ‘targeted’ Other Guys were ‘reached’ which is the obvious first and crucial stage in the Cascade. If they don’t get it, then it can’t change them, right? Facebook said that by their standards of evaluation, the Other Guys got the message enough to know they got the message and actually processed it. Thus, all the Other Guys took at least one trip down the Communication Cascade according to Facebook.
Now. Here’s the good science part. After randomizing Other Guys to Facebook treatment or control, the researchers then called a large random sample of these Other Guys and did a telephone survey to assess the effect of the Facebook ads.
To assess the impact of the ads, on Saturday, October 13 through Monday, October 15 the polling firm AMM Political Strategies completed live interviews with 2,984 individuals on the voter file (all of whom had associated phone numbers). The firm called the numbers in random order and did not have access to the treatment assignment status of the respondents.
What questions did they ask?
The questionnaire, given in Appendix 2, asked respondents (1) whether they had a positive, negative, or no impression of the collaborating candidate, (2) whether they had a positive, negative, or no impression of the opposing candidate, (3) their vote intention in the upcoming election, (4) whether they recalled seeing any ads for the candidate on the internet, and (5) how often they used Facebook over the last week.
Simple. Direct. To the point. This is what you need to know. You could get more sophisticated and design a Standard Model questionnaire to test persuasion theory, but, hey, this is practical persuasion in the real world and in real time. You need key information right now and this telephone survey delivers the goods. So what are the goods? Here’s the key table of results for the first field experiment.
The table reports the regression coefficients and their confidence intervals that compare the difference between treatment and control conditions. Across the top right you see the main outcome questions (impression, heard of, use FB). On the left column you see the sample broken out into two types: All the Other Guys or just the Other Guys who self reported FB use.
MP1. Note my chicken scratches by those results in brackets. Those are the 95% confidence intervals around the mean difference between treatment and control. Notice that all, and I mean all, of those bracketed 95% CI intervals range from a negative to a positive which means they include zero which means on a sample size of either 2948 or 1364, the Facebook treatment had not even a statistically significant effect. Nada. Nothing. No change on name recognition. No change on favorability for either candidate. No change on voting likelihood. Zero. Zed. And Zed is dead, baby. Zed is dead.
Even if you cannot run a regression yourself and need help reading the results, this table and those confidence intervals shout that the Facebook ads which ‘reached’ every ‘targeted’ Other Guy made no difference on any key outcome. And with these large sample sizes a Tooth Fairy Windowpane would be detectable.
MP2. Note that the null effect holds for both the entire population of Other Guys and the subset of Other Guys who self report using Facebook. In medical research, that entire population is called Intention-To-Treat, meaning that you include everyone you aimed at even if you didn’t hit them because that’s what the real world requires. If you are running for office, you are Intend To Treat all the voters with your treatment. Facebook has got to deliver something in that reality and in this Local, Facebook delivered nothing.
Now, Facebook would steer you to that smaller subset of Facebook using Other Guys. Hey, how can we change Other Guys if they don’t use Facebook? Two replies to that point. One: As the table proves, Facebook ads didn’t change the Facebook users! Two: Word of mouth, baby; I thought that Facebook was the Cool Table and if you convinced the Cool Table, the Cool Table would influence the great unwashed unDigirati.
And as bad as the outcomes are for the first experiment, they are just as bad for the second experiment. Raise Zed from the dead of study one, then bury Zed again with study two.
There’s a ton of theory and research nuance in this paper and if you are this kind of propeller head, please read the paper. But for persuasion panthers, the nuance is noise. Absolutely nothing of practical and favorable impact came from Facebook advertising. The researchers combine the nuance and noise nicely in this paragraph.
To change merely 20 voters’ minds out of the roughly 20,000 who were exposed to the ads would have rendered the $200 ad buy fairly cost effective (at $10/vote). However, to reliably detect the implied 0.1 percentage point effect (20/20,000 = 0.001) of such an ad would require an experiment of roughly 5 million voters. Our experiment (and indeed the legislative districts we studied) contained far fewer than 5 million individuals and could not have detected effects of this miniscule size. As with commercial online advertisements, understanding whether very cheap political online advertisements are cost effective will likely remain ‘‘nearly impossible’’ (Lewis and Rao 2012). At the same time, our experiments do cast doubt on the view that online advertising has substantively meaningful impacts on political attitudes or electoral outcomes.
Broockman and Green thread the camel through needle by noting that it is theoretically possible to create a real voting change and at a cheap price, but it would be practically impossible to detect the change because it is so small. Stated another way, if you did this experiment for a candidate running for President, you could cheaply run a Facebook ad campaign that would truly change a few thousand votes in the 100 million cast and if you could afford the cost of a random sample large enough to detect this true change, then, yeah, Facebook advertising works.
We saw this same contradiction with the infamous CDC statistical science on their wildly trumpeted anti-smoking media campaign. While they claimed 200,000 quits, the data they collected in no way proved that because they didn’t survey a random sample of several million smokers. Even 200,000 is a trivial number against a population of 43,000,000 and you need a huge random sample to demonstrate it. The CDC just faked it with a small study and then extrapolated the small (and biased convenience) sample results to reality. When the only Other Guy who counts is the President who hired you, then you can say things like this in public and not get fired. But, if you are running for office or running your Mom and Pop business, this is how you get motivated to find a new line of work.
Contrast that persuasion statistical science with this mere science. Pick a couple of different Locals. Run the ad campaign as Facebook is designed. Then draw large random samples of your Other Guys and survey outcomes. This research actually designed a rational way to count the change and finds, alas, no change to count.
If you are a Facebook panther, you know how to respond to this. Only a couple of cases. Elections are weird. Try this with more races and see what you get then, baby. And, hey, you need to collect larger samples!
Which is another way of saying that Facebook persuasion is a Tooth Fairy Tale. Sure, collect stupendous sample sizes, adjust for hundreds of different variables, run thousands of different regression equations, then find the asterisks and holler Significant At Last, Significant At Last!
But realize that you are not doing what Facebook says you can do easily and cheaply: Change a lot of Other Guys.
You can put your messages in front of a billion Other Guys and do it very cheaply – same with Google and AdSense. These iAd wizards sell exposure for fractions of a penny. But where’s the change your mille purchases?
The public argument for iAds is that they do work, but when you get down to details like this, you rarely find that iAds work as simply and directly as advertised. You need nuance, like this.
Advertisers use Facebook and Google for different reasons. Google is effective when people search for things they already want to buy and then see ads for relevant products. Facebook is better for educating consumers about products they might want to buy in the future.
“The smart brands learned to use Facebook to create demand, and Google to fulfill demand,” said Bob Buch, chief executive of social advertising company SocialWire.
Ohhhhh! Facebook ads educate Other Guys then Google ads make Them hit the TACT. Yeah, baby, that’s the ticket.
Where’s the data on that one?
What’s funny about this is the incredible Dissonance almost everyone involved is experiencing. Facebook and Google are obvious persuasion machines that laser Other Guys then Box ‘n Play them with astonishing variety, speed, and constancy. They can’t miss.
Yet, when you count the change, the change almost never counts, and if it does, it requires a Tooth Fairy to find it which destroys the initial belief that Facebook and Google are persuasion machines. Good grief, if iAds are that good then why do you need an epi propeller head from the Harvard School of Public Health to invent, oops, discover it?
You see the klonger. People believe so strongly in the persuasion that sold Facebook as a persuasion machine that they cannot accept the reality Facebook returns. They love Facebook as a persuasion machine and the more they suffer Fairy Tale results, the more they love the Facebook persuasion machine.
David E. Broockman and Donald P. Green. (2013). Do Online Advertisements Increase Political Candidates’ Name Recognition or Favorability? Evidence from Randomized Field Experiments. Political Behavior.
P.S. Again, kids, it’s all about WATTage. Facebook, Google, or almost any standard iAd platform is a conflicted information environment where the receivers (the Other Guys) are using the platform as much as the platform is using them. No one has figured out how to control the WATTage of the Other Guys for the specific persuasion message running in the iAd.
P.P.S. So why is everyone using iAds, Professor Poopypants?
Hey, have you already forgotten about selling sand and ice in the service of Dissonance?
It’s a persuasion machine as long as a panther persuades you it is a persuasion machine, not because independent testing proves it is a persuasion machine. People are eating the persuasion from the digital page called Facebook.