Always Be Testing: Detecting Novelty in A/B Tests
ATTENTION: You must signup in advance to attend this workshop. You will need to have a valid SXSW badge, and an activated SXsocial account. To reserve your seat, please go here: https://sup.sxsw.com/schedule/IAP25757
In the Obama campaign's digital fundraising operation, we constantly A/B tested all aspects of our emails. Sometimes we would stumble upon surprising results that gave us statistically significant increases in our donation rates -- and then, just as quickly, those effects would wear off. From special characters in subject lines to ugly yellow highlighting on entire paragraphs of an email, we found plenty of tactics that displayed classic "novelty effect" patterns.
This two and a half hour long workshop will demonstrate how to detect those novelty effects through multi-stage A/B testing, long-term testing, and re-testing -- and how to avoid falling into the "best practices" trap of thinking you're done after one successful round of tests.
After introductions, we will have a brief review of why testing is so important in the first place, with plenty of real-world examples. We'll also quickly go over the basics of A/B testing for digital media such as email, web, and ads.
Next, we will look at some examples from the Obama campaign of messages and tactics that tested well, but eventually lost their novelty. After discussing common-sense ways to anticipate potential novelty effects, we'll dive into various methods of multi-stage and long-term testing you can use to detect them empirically. We'll also talk about the art and science of knowing when to return to old test results and re-test.
The training will also include plenty of time for questions and answers, brainstorming, and fun anecdotes from the Obama campaign.
- Participants should be familiar with basic principles of randomization and choosing representative samples, as well as standard metrics such as open rate, click rate, donation rate, webpage conversion rate, etc. However, we'll still have a quick review of A/B testing principles, so even novices should be able to keep up.
What to Bring:
- No equipment is necessary.
Digital & Quantitative Consultant
Amelia Showalter LLC
Amelia Showalter is the former Director of Digital Analytics at President Obama’s re-election campaign. She led a cutting-edge Digital Analytics team that designed and implemented hundreds of experiments to improve the performance of all types of digital outreach. Before the Obama campaign, she provided data analysis and microtargeting to dozens of campaigns as a strategist at Changing Targets Media and MSHC Partners. Her new mission as a solo consultant is to bring the Obama campaign’s culture of rigorous testing and analysis to progressive organizations, campaigns, and firms.