What Do We Mean When We Say “Experiment”?

In a lot of our materials, and in a lot of the buzz out there about us, you hear talk about “experiments.” This blog post is about what these experiments are, and why they are very valuable for campaigns. Specifically, we’re going to talk about the randomized controlled experiment (RCE).

Value Proposition:

The rationale for doing an experiment is simple. Why spend $1M all at once hoping something works, when you can spend $15k to figure out whether that $1M has any shot of working? Better yet, why spend $1M hoping something is going to work when you can spend $50k trying 4 different things, and select the one that works the best? Experiments allow us to test our ideas at small scale, and scale up those ideas that we measure actually generating a result for the campaign.

You Making This Stuff Up?

Actually, we didn’t invent this idea — many fields use it. For example, experiments have been in use for decades in medicine. “Wait,” you say — “doctors aren’t allowed to experiment on live patients!” As it turn out, they are — they are called FDA drug trials.

As an example, let’s say a medical doctor has figured out this really cool medicine that he believes can help reduce the risk of heart attacks.

After doing a whole bunch of successful smaller tests in the lab and with animal subjects, the FDA clears the doctor to do a randomized controlled experiment to prove the true benefit of the drug. The doctor finds a whole bunch of test subjects who could be at risk for heart attacks, and enrolls them in his study.

Then the doctor randomly divides his patients into two groups. He then does some checks to ensure the randomization has worked correctly across some significant factors. For example, he may check to ensure that age distributions are balanced between the two groups. He may check to ensure high blood pressure rates, diabetes rates, and prior heart attack rates are balanced between the two groups. Eventually, he has two statistically similar groups, which we’ll call treatment and control groups.

Now that the doctor has two statistically similar groups, he administers the “control” treatment, a water pill, to his control group. The other “treatment” group gets the medicine (the treatment) that the doctor thinks may help prevent heart attacks.

If the group that got the medicine shows a statistically significant improvement over the group that got water pills, the doctor can scientifically say that he’s got a winner on his hands.

We Can Do The Same Thing In Politics. In Fact, Our Firm’s People Have Around 500+ Times. 

Let’s say you are a Republican campaign. The Dems are doing their “War-on-Women” shtick via a mailer, and you want to respond quickly. You decide you are going to respond by mail, but you don’t know what creative to put on the mail piece. Some of the staff on the campaign say you should have the candidate’s mom and daughters on the mail piece, talking about how the candidate has always stuck up for women. The candidate’s wife says the mail piece should be the candidate straight-to-camera, will bullet points on how he worked to strengthen women-owned small business incentives. The mail vendor thinks it all about having the former Republican female governor on the mailer, talking about your candidate’s pro-woman record. Instead of guessing, and hoping for the best, you test all three ideas.

Out of the 300,000 women you want to convince, you select out a sample of 40,000. You randomly divide these groups into four groups of 10,000, and make sure factors that could influence the outcome of the test are balanced (things like age, party registration, geography, etc). Three of the groups each get one of the designs your team thought up (from above).

One group gets nothing — no mail.

After the mail is delivered, you call in and conduct voter-ID calls within each group. You compare the performance of groups. You understand where support was in the entire population before you did anything by looking at the control group’s candidate support rates, and what percentage “lift” over the control each mail piece generated. Brace yourself, because a lot of mail pieces aren’t going to do any better than the control (those who got no mail), but some will. Now you get to act with fresh knowledge in your hands, and you can select the mail piece that you know will work best with this population of voters.

We’ve done this example for persuasion mail because it’s easy to conceptualize. The good news is, you can do these sort of measurements for most any form of voter contact. (TV? – Yup      Volunteer Door Knocks? – Those Too     Digital Pre-roll? –  Yea, Even Those).


Stop guessing. Start acting from knowledge. Your competition (the Dems) is already doing it. They don’t spend their millions unless they know its going to work. Can you afford to keep guessing?

Interested in discussing more? Contact us.