In order to continually improve the customer experience the Netflix product, design and algorithm teams have run over a thousand A/B tests with tens of millions of users. We have had many successes and, even more importantly, many more failures. If too many of our tests are wins then we’re not pushing hard enough and not learning enough.

For years we made steady progress but 2010 was a turning point. As we started doing large scale UI testing on the “ten foot experience” we learned a smarter way to do A/B testing, rapidly accelerated the pace of product innovation.

Todd Yellin will share an assortment of design and algorithm wisdom gleaned from getting at the truths that can only be gained from focusing on actual user behavior. He’ll share some less obvious lessons about interaction design gained from some supposedly sure bets that flopped. And other more successful tests run on the interfaces and algorithms from sign up to content discovery to actual viewing experience.