A marketer’s ultimate goal is to maximize conversions, turning their efforts (designing a website, creating email campaigns, allocating advertising dollars) into actual profits. A/B testing is a well-known tool for helping companies identify the option that can generate more customer action, but is it really the best tool for optimizing conversions?
At a high level, A/B testing is easy to understand. You have two options, A and B, and they are presented in equal amounts. The version that generates more conversions (clicks, signups, purchases, etc) is the winner. Marketers then implement the winning version to all subsequent users. A/B testing works, and there are plenty of success stories out there touting its excellence; both Obama and Romney used A/B testing to boost donations during their Presidential campaigns in 2008 and 2012.
Traditional A/B testing
Sure, Presidential candidates use it, but A/B testing isn’t perfect. The reasons:
1. A/B testing can be slow.
A/B testing strives for statistical significance. As a result, a sample size for each experiment is predetermined to make sure the results are not merely due to chance. However, it may take a while for an experiment to obtain the necessary number of subjects. For example, if a test requires a total sample size of 10,000 visitors, but you only get 100 site visitors each day, then you will be waiting over 3 months for your experiment to be finished.
2. A/B testing doesn’t maximize conversions.
Running an A/B testing experiment may actually be detrimental to your conversion rates. Suppose you are testing two options: A, which is highly effective in generating conversions, and B, which is a total dud that no one ever chooses. Every time you show option B, you’re missing out on a potential conversion that A could have created. For an experiment with a large sample size, the unrealized profits could be huge.
For example, suppose you are testing 2 options and need 3,000 total trials to obtain statistical significance. It turns out, A is awesome and earns you an average of $5/trial, whereas B only earns on average $1/trial. Thus, every time you showed choice B, you lost $4 of potential revenue in which A could have earned; for 3,000 trials, that’s $6,000!
Multi-Armed Bandit
So how do you minimize losses associated with testing? Enter the multi-armed bandit (MAB) algorithm, a clever system that learns and adapts to save time and maximize conversions.
The term multi-armed bandit comes from gambling. Imagine you are standing in front of a row of Las Vegas slot machines with a limited amount of time and money. You know some machines have higher payouts than others, but how do you divvy up your resources to maximize earnings? You play each slot machine once and after each round, you decide which machine to go to next depending on the winning or losing result, eventually identifying the highest-paying machine.
Likewise, marketers can use the MAB algorithm to optimize their conversions. First, every option is tested. If A generates more conversions, the algorithm will direct more users to A. However, if A starts to slip in conversions, then the system will revisit the other options and direct more visitors there, thus eliminating bias. You can monitor the progress of each option to declare a winner or add more options to continuously test, improve and strive for even more conversions!
Now let’s go back to the previous example, but conduct a MAB experiment, rather than an A/B test. The MAB algorithm would notice that A is a better performer than B much sooner than 3,000 trials. As a result, it will direct more users to A, optimizing your earnings and minimizing the opportunity cost of testing.
A/B testing works, but it’s not efficient if you want to be flexible by adding testing options on the fly, or want quick results. Think of it as the dumb beast that naps while its experiment is running, waking up only when the tests are finished.
MAB, on the other hand, is an adaptive machine, determining its next move based on the results of the previous round. It immediately optimizes your conversions rates and allows you to continuously test with no waiting periods. Marketers previously stayed away from MAB because of the misconceptions that it is far more complicated than A/B testing when in reality, it’s much simpler because you don’t have to worry about determining a meaningful sample size. Furthermore, MAB experiments have a fire-and-forget and continuous nature, so you don’t need to wait for one experiment to finish before you add another option for testing.
Questions about MAB? Leave them in the comments below.
This blog post is a modified version of the original post from the Iterable blog.