Why You Need to Combine A/B Testing and Personalization

A/B testing has been a must-have for data-driven marketers for decades. With A/B testing, you stop making guesses about your digital experiences. Instead, you run tests and use real numbers to tell you what works and what doesn’t.

But I would argue that traditional, one-size-fits-all A/B testing is a thing of the past. Let me take you through a simple, theoretical example to show you what I mean.

Traditional A/B Test

Assume you’re testing two different website experiences. We’ll call them Experience A and Experience B. If you ran a traditional A/B test, you may see results like this:

Let’s assume the Y axis represents an important KPI to your business such as revenue per user, conversion rate, sign-ups, clickthroughs, etc.

These results tell us that Experience A clearly performs better than Experience B. Knowing this information and assuming the results are statistically significant, you would set 100% of your traffic to see Experience 1 and move on to your next test.

But let’s also assume that you have two audience segments such as new visitors and return visitors. Let’s call them Segment 1 and Segment 2. Each segment is of equal size, and each segment responds differently to the tested experiences. If you broke down your test results by segment, you may see something like this:

These results show that Segment 1 responds better to Experience B, while Segment 2 responds better to Experience B. This is something we wouldn’t know from looking at the results shown in the previous graph. We couldn’t see this from the original results, because we were looking at an average of how each segment responds to each experience. Both Segment 1’s and Segment 2’s reactions to Experience A and Experience B are averaged together to produce your overall performance numbers for those experiences:

Clearly, that average hides some really interesting differences in the preferences of these segments.

The Cost of Picking a Winner

Without personalization, even if you broke down the test results like this, you would still end up picking Experience A as the winner. After all, if you have to pick one single experience that is best for all, Experience A would be it. But by selecting Experience A, you’re missing out on the additional benefit of showing Experience B to Segment 1.

With basic segment-based personalization, however, you wouldn’t need to pick one experience over another. You can pick Experience A for Segment 2 and Experience B for Segment 1. In other words, you can choose the experience that “wins” for each segment, not just the experience that wins overall.

Here’s what it would look like if you stacked the results of the optimal experience for each segment compared to the optimal experience for all:

Testing in the Real World

This was, of course, a highly simplified example. In reality, there is an unlimited number of ways you can segment your audience, and your segments will naturally vary in size. But the important thing to take away from this example is that just because a test produced a winning experience, it doesn’t mean that experience is the best for everyone. It just means it’s the best you can do without personalization.

All of your traditional A/B tests are likely hiding critical information like this simplified example showed. It doesn’t make sense to miss out on unrealized benefits when you could just combine your A/B testing with personalization to deliver something relevant to everyone.

But let’s not forget the big picture. It’s not just about producing better results for you. Showing more relevant experiences to everyone ultimately leads to a better experience for your customers and prospects. A personalized experience often helps a person feel remembered and valued. In most cases, when done well, a person will not know that an experience has been tailored to him; it will just feel more relevant. For example, if you are a B2B company and you change your homepage hero messaging dynamically to explain how your company provides value to each visitor’s industry, it is much easier for your visitors to determine if your company is relevant to their needs.

How to Incorporate Personalization into Your A/B Testing

With years of A/B testing under your belt, it may seem unnatural for me to suggest doing it any other way. But I’m not suggesting that the overall principles of A/B testing don’t still apply in a personalized world. There are just a few adjustments you need to make.

Rather than running tests to come up with one single best experience for all visitors, identify your most important segments and test different experiences for each one. B2B companies may segment by industry or company size (small business, enterprise, etc.). B2C retailers may segment by favorite category or preferred gender. Financial services companies may segment by products used (credit card, auto insurance, mortgage, etc.) or stage of life (student, new homebuyer, etc.). You can test several experiences for each of your key segments to find the experience that works best for each one.

And any time you run a test, always filter the results for different segments. You may find that a particular rule-based experience is yielding good results, but if you dig deeper, you may see that it performs better for returning visitors than for first-time visitors, for example.

While rules can help you manually target different experiences to different segments, machine-learning algorithms can automate the process, enabling you to better target experiences at the segment — and even individual — level. With many segments to optimize for – including ones you can’t foresee – using rules can become labor-intensive, and optimizing at the individual level is basically impossible.

Final Thoughts

In the early days of A/B testing and conversion rate optimization, finding a single winner of a test was game-changing. But today, you don’t need to pick just one experience to display to everyone. You can personalize experiences to different audiences or even to different individuals. There’s no reason to cater to the lowest common denominator anymore. And with the application of machine learning, you can turn A/B testing on its head.

Read more:

• Ways A/B Testing Can Make Gap Inc. Millions