Persuasion science uses statistics.  Some people want to run away screaming anytime math enters the equation, but I want to show you a way of using numbers in a simple and straightforward way that helps you understand what’s really happening.  It’s called the “effect size” and it answers a simple question:  How big of a difference is it?  An outstanding psychologist named Jacob Cohen articulated this concept.  Another excellent psychologist, Robert Rosenthal also developed the idea further.  I’m going to combine their ideas and put them in the Windowpane Display.  Here’s how the Windowpane works.

Think about a window.  Imagine that it is divided into four equal panes.  Easy to visualize, right?  Those 2 by 2 panes.  Now, let’s say we do an experiment where we randomize one group of people to get the New Thing while another group of people get the Old Thing.  To make the math simple, we’ll give 100 people the New Thing (treatment group) and 100 people the Old Thing (control group).  After each group does its Thing, we carefully observe each person to see if they Changed.  Either they did Change or the did Not Change.  Let’s dress up the Windowpanes with these labels.

WINDOWPANE WITH TREATMENT and OUTCOME

Pretty simple so far.  We’re testing the New Thing against the Old Thing.  We have 100 people randomly assigned to each group.  We then see how the people Change either into Yes or No.  Now, let’s fill in each of the four little windowpanes to demonstrate different scenarios.

We’ll start with failure which is what usually happens with science.  All our good intentions are smashed against the Rock of Experimental Science and nothing happens.  Let’s be polite and call this the No Effect outcome rather than use the words scientists use when looking at the stat results on the screen and realizing their next grant application just died.  It looks like this.

NO EFFECT

We’ve got 50 people in each little windowpane.  To understand what’s happening, read each row.  We started with 100 people in the treatment condition who got the New Thing and when we observed them we found that 50 of the 100 changed and 50 of the 100 didn’t change.  We also started with 100 people in the control condition who got the Old Thing and when we observed them we found 50 of the 100 changed and 50 didn’t.  No effect.  Nada.  Zip.  The New Thing is not different from the Old Thing.  [Side Bar:  If you’re a stat maven or just pretty quick you know that failure would result if both rows were 10/90 or 30/70 or even 90/10, anything as long as both rows have the same percentage.  Failure is not just 50/50, but rather when both rows show the same finding.  I used 50/50 because it makes the math and the concept easier to follow.]

Now, let’s create an example where we start to get differences.  Let’s assume that Something Happens when people get the New Thing and it looks like this.

SMALL EFFECT

We now see on the rows and the columns, a 45/55 effect, a 10 point difference.  In social science parlance, this 10 point difference is called a “small” effect as popularized by Jacob Cohen in his work on power analysis and effect sizes.  Make sure that you see the impact of the treatment.  Notice in this example that more people who get the New Thing showed the desired change (read the row) compared to people who got the Old Thing (read their row).

Now, let’s increase the effect size.  Let’s go from “small” to “moderate.”  Here’s the Windowpane for a moderate (also sometimes called “medium”) effect.

MODERATE EFFECT

Now, our row values are 35 and 65.  A moderate effect is a 30 point difference.  That sounds somewhat impressive, a 30 percentage point difference.  Think about this medium effect another way.  Notice that 65 is almost twice as large as 35.  A medium effect means that you’re getting almost twice as much change in the treatment group compared to the control group.  A moderate effect is getting to be pretty obvious.  Think how obvious a “large” effect must be.  It looks like this.

LARGE EFFECT

The row values here are 25 and 75, a 50 point difference.  Now the rate of difference is three times with the Treatment producing a 300% increase over the Control.  That’s big.  That’s obvious.  Take a quick scan now and review the four Windowpanes, No Effect, Small Effect, Medium Effect, and Large Effect.  See the numbers change.

Rarely will you see reports offering something like the Windowpane.  Instead you’ll see correlations (r), standard deviation effects (d or g), or ratios (risk, absolute, relative, odds).  Here’s a handy conversion guide again using the Cohen conventions.

I also caution you to go High WATT when reading those odds ratios.  They are extremely persuasive numbers than can be played like chords on a guitar to produce different tones for the same values. For example, a risk that is found to be “150%” greater than the control group sounds like a big deal, but it is only a Small Effect.  Just scan that range from 1.5 to 4.0 through 9.0 to get a sense of scale.  Sure, “150%” sounds big, but compared to “400%” or “900%,” not so much.  And most health and safety research published in good peer review journals finds these Small Effects, so when you see “134%” increase, there’s a natural temptation to go “Wow!” when the effect is barely detectable over random variation.

The Cool Table guys know exactly how to play the statistical sophist with  ratios and make the weaker Argument appear to be the stronger Argument.  Cassandra speaks!  You are warned!

The point of this demonstration is to show that you can think with numbers in a practical and efficient way without having a statistician in the room.  Anyone can handle the windowpane approach with numbers.  Just have a clear definition of Changed? (Yes or No) and a clear definition of the Group (Treatment or Control).  Then just count and look for percentage differences.  A 10% difference is small, 30% is moderate, and 50% is large.  And, realize that while “small” may be hard to detect, it can definitely make big practical effect.