ATTENTION: You must signup in advance to attend this workshop. You will need to have a valid SXSW badge, and an activated SXsocial account. To reserve your seat, please go here: https://sup.sxsw.com/schedule/IAP25757

In the Obama campaign's digital fundraising operation, we constantly A/B tested all aspects of our emails. Sometimes we would stumble upon surprising results that gave us statistically significant increases in our donation rates -- and then, just as quickly, those effects would wear off. From special characters in subject lines to ugly yellow highlighting on entire paragraphs of an email, we found plenty of tactics that displayed classic "novelty effect" patterns.

This two and a half hour long workshop will demonstrate how to detect those novelty effects through multi-stage A/B testing, long-term testing, and re-testing -- and how to avoid falling into the "best practices" trap of thinking you're done after one successful round of tests.

After introductions, we will have a brief review of why testing is so important in the first place, with plenty of real-world examples. We'll also quickly go over the basics of A/B testing for digital media such as email, web, and ads.

Next, we will look at some examples from the Obama campaign of messages and tactics that tested well, but eventually lost their novelty. After discussing common-sense ways to anticipate potential novelty effects, we'll dive into various methods of multi-stage and long-term testing you can use to detect them empirically. We'll also talk about the art and science of knowing when to return to old test results and re-test.

The training will also include plenty of time for questions and answers, brainstorming, and fun anecdotes from the Obama campaign.

Prerequisites:
- Participants should be familiar with basic principles of randomization and choosing representative samples, as well as standard metrics such as open rate, click rate, donation rate, webpage conversion rate, etc. However, we'll still have a quick review of A/B testing principles, so even novices should be able to keep up.

What to Bring:
- No equipment is necessary.