Campaigns and issue-advocacy efforts usually spend the lion’s share of their money on TV ads. Very few campaigns test ads to ensure that they will work on their intended audience before they air them. Of those that do test ads, many employ focus groups or dial tests to elicit feedback about their ads. The problem is that focus groups (and their dial-test cousins) are not very good measurement tools. These “tests” are typically nothing more than an 11 (or so) sample with the propensity to produce highly questionable conclusions.

The Problems With Relying On Focus Groups

1. Hawthorne effect: People who know they are being studied/observed act differently then those in more natural circumstances.2. Priming effects: We start focus groups by telling participants that they are here to talk about an election, election ads, and how those ads influence their opinions. Unfortunately, framing the activity this way primes them to pay more attention, take positions, debate those positions, and reflect upon whether their own opinions are shifting. Unfortunately, this is not the mind-state in which most people normally watch commercials. The reality is that a campaign’s commercial is going to be played in between a Jeep Wrangler commercial and a Charles Schwab IRA commercial, all while the kids are playing in front of the TV. After this series of 30-second commercials has ended, the viewer will be thrilled that his or her show is back on. The average viewer does not spend 10 minutes thinking through what they just saw.3. Self-selection bias:The type of person willing to drive across town and spend 2-3 hours away from their kids on a school/week-night for $25-$100 is probably not representative of the larger electorate as a whole.4. Geographically nonrepresentative: A given focus group can only draw people from one geographical region, usually an urban one. This means that state-wide candidates often only receive feedback from a geographically skewed portion of their electorate. Ads that work well for urban voters do not necessarily work well for rural voters.5. The moderator effect: Different focus group moderators will lead group discussions in different directions, whether consciously or unconsciously, by employing various verbal and nonverbal cues. Group moderators have incredible power to influence the topics that are discussed and the conclusions which are reached. When a measurement’s outcome is dependent on the individual running the measurement and cannot be repeated by others, it is referred to as pseudo-science (right up there with telekenisis or fortune telling).6. Dominant respondent bias: In any group, some individuals have stronger opinions than others. When you gather participants around a table, the group as a whole tends to defer to those who express the strongest opinion, regardless of whether it is a shared opinion. In real life, people do not typically absorb ads next to someone with a strong opinion helping to shape their perceptions. The influence of dominant participants undermines the ecological validity of focus groups and skews their conclusions in unpredictable ways.7. Laughably underpowered: You wouldn’t believe a pollster if he took a poll of 11 people, so why on Earth would you believe 11 people’s opinion about your ad? The reason they call focus groups ‘qualitative research’ is because the statistical laws that allow us to generalize ‘quantitative research’ don’t apply to such small samples.8. Bonus – if we add a dial it must be better… right? Nope, still qualitative pseudo-science: Adding a dial and the instruction to “turn left if you like what you are seeing, turn right if you don’t like what your seeing,” makes for some cool graphs and the feel of quantitative measurement. However, all of the same biases and statistical limitations still apply to the new, fancier measurement data. We are still taking people out of the conditions they will see the ads in, priming them to over-rationalize, opening ourselves to strong self-selection bias, and subjecting our participants to the distorting influence of both moderators and dominant participants. With dials, focus groups may be able to provide fancy graphs, but in the end, they are still underpowered, small-sample studies tracking something we don’t really care about. After all, we don’t care if people like or hate an ad, we only care whether an ad is able to influence how someone is going to vote.

The Good News

The good news is that there exists quantitative, representative, empirically sound methods for testing ads. At Øptimus, we have developed, deployed, and changed the course of statewide elections using these quantitative methods. If TV ads are going to be the largest amount of spend on your campaign, shouldn’t they work?

Get in contact with us to figure out which ads work, and which ads don’t.


About the Author:

Brian Stobie

Brian is one of three founders at Øptimus, a data & analytics consultancy based in Washington, D.C.

Brian is passionate about working for individuals and groups that share his beliefs, and that are open to using data, analytics and insight to run disciplined and highly productive campaigns.

Brian graduated from the University of Texas at Austin, where he developed a severe college football addiction and majored in electrical engineering.


Related: