5 Things People Get Wrong About A/B Testing and How to Fix Them

Data and statistics don’t lie.

But when it comes to between gut feeling and numbers, most people will go with their gut and forego numbers.

That being said, it must also be remembered that facts are stubborn things, but statistics are pliable.

So what do you do?

The key is to interpret the data and numbers holistically.

The power of numbers and data is best illustrated by Split testing. Split testing is one of the most useful ways in which you can direct traffic to your website and increase conversion rates.

It has been shown that A/B testing generates up to 30-40% more leads for B2B sites, and 20-25% more leads for ecommerce sites.

However, it is easy to trip and commit mistakes which could prove costly to your brand. Avoiding them will maximize the effectiveness of A/B testing and yield significant results. Let us examine the 5 common mistakes and how to avoid them.

Ready?

Let’s start.

Not collecting enough data and being impatient

Obviously, we cannot test our hypothesis on the entire population, so we choose a sample size and run our testing accordingly.

So you take a sample size, run the tests, get the results and make a decision.

Sounds great right?

NO.

This is one of the most common mistakes made in split testing.

It has been proven statistically that the more data we have, the more confidence we can have in our estimations, the lesser is the margin of error and greater is the precision.

After all, you cannot generalize the behaviour of the entire population based on your sample size and results.

Another mistake most people make is not to wait for the test to yield statistically confident results.

Let us take an example. Suppose you are testing a contact form with fewer fields in order to generate more leads.

Which version is more successful?

While it is tempting to go with version 2 because it had twice as many form submissions as version1, it is not the answer.

Look at the sample size. Just 66. Not even 100.

So what is the verdict?

The data is not enough.

So how do you fix this problem?

1. Decide on a sufficiently large sample size for testing
2. Decide how long the test should be run. Resist the urge to stop testing at the first statistical

significance.

Takeaways:

1. The test should be run until variable being tested achieves at least 95% confidence levels. Then and only then can you terminate the test.
2. While the test may take many days, always ensure that the test is terminated on the same day as it was begun.

Testing more than one wrong thing at a time

It is easy to get excited about split testing and rush to test everything. You change the headline, testimonials, buttons, dropdown menus and then get results that leave you more confused than before.

Maybe the headline is dragging conversions down while the buttons may be increasing the conversions. The bottomline is you don’t know which variable is affecting the test result in what way.

Often you are not even sure if what you are testing will have a positive impact on conversions.

Check this example for instance in which two different popups are tested.

Both the popups are completely different. The format, the color, the headlines all are completely different.

Even if you get a clear winner, you can’t be sure of the factors which contributed to its winning.

So hold your horses.

Test one thing at a time.

So how do you start? Here are some actionable tips:

Remember what can be measured can be understood.

1. Remember what can be measured can be understood.
2. Start by measuring your website’s current performance using analytics.
3. Define your website goals in terms of actions. Eg. Increase clickthrough rates, decrease cart abandonment rates, etc.
4. Define your KPI’s and segment in terms of behaviour and outcome.
5. Prioritise important pages in terms of traffic and conversions.
6. Frame a hypothesis on one element and test it.

There are several elements you can test. You can test headline, content , buttons or CTA’s.

So the question arises what do you test?

It is here that defining website goals and KPIs becomes important. This is also the reason why we recommended that you measure your website’s current performance.

This will enable you to focus on the area of weakness and thus narrow down on the data you want to test. And narrowing down on the data you want to test helps you choose the variable and the hypothesis accurately.

Another alternative is to do multivariate testing (testing and comparing one element at a time). It involves comparing a higher number of variables and getting more information on how these variables interact with each other,

Ignoring external factors

Here, in this context, we are talking specifically of one thing: seasonality

Take a look at this graph

Would you test during the spike period as pointed?

If you said Yes, you might reason that traffic is flowing and the test results would probably be obtained faster.

But the fact is it would be a terrible idea to test during the spike.

For instance, if you are testing while running an ad campaign or during a special offer season during holidays, the test results will be majorly skewed.

Because data always changes, more so during such periods and you do not want such external factors to drive major changes in your implementation that would not work for the rest of the year.

So it is recommended not to test during such times so as to have maximum control on your testing and thus results.

How do you identify at what time to test?

1. Examine your past. See for the spikes in traffic and stay away from testing during those phases.
2. Look to your future. Do not test when you expect a major spike in the coming time.

Ignoring small gains

This is more of an instant gratification problem.

We want results that show gigantic increases every time we test.

Hold on.

Let’s not romanticise A/B testing. Unless your website is terribly designed and wrong, chances are you will see improvements which are less than 20% ( at the maximum).

Don’t ignore them.

A/B testing is rather like weight loss. You focus on gaining something little by little until it all compounds to the result that you want over a period of time. (Mostly long periods of time)

Let us assume we test a hypothesis and we see an improvement of say 2%. Let us also assume that you receive 100,000 visitors.

How much does it translate to?

That’s around 2000 visitors you would not be reaching to should you choose to ignore the test results.

Do you want to ignore 2000 visitors? I guess not.

Misreading results

This happens when you are not doing enough maths.

Sometimes we focus so much on conversions and clicks that we lose sight of the bigger picture.

What is the goal of A/B testing? To improve conversions.

What is the goal of conversions? To generate revenue.

You may be testing properly and you may even get conversions. But what if increased conversions is hurting your revenue?

Let’s illustrate this with the help of an example.

Let us say you have 3 pricing plans for your product

1. A free plan
2. A basic plan for $25
3. An advanced plan for $50

Let us say you want to split test by reducing the price of the basic plan to $10.

You rush to test and the conversion is 4%. Hooray !!!

And you are in a even bigger rush to implement the changes.

Hold on.

Let us assume the pricing page sees 500 visitors per day. It converts at 23% with a A/B of

1. Free at 17%,
2. Basic at 4%
3. Advanced at 2%.

That’s $750 in revenue per day.

Then there’s the test where the Basic price is lowered to $10. We’ve got a 4% increase in conversions. Let’s say Free picked up a percent and Basic plan picked up 3%.

Unfortunately, 1% of Advanced plan users thought the value at Basic plan was too good, so they switched over to Basic.

Now you’ve got an A/B of Free/18%, Basic/8% and Advanced/1%. That’s a total of $649.95.

So conversions went up, but you would lose $100 per day.

So the key is to look at all the parameters while judging the results. Which means you have to do you math properly.

Conclusion

Now that we have examined the 5 common mistakes of A/B testing, you should keep them in mind when running your future tests.

A/B testing is an effort to be objective in making decisions. Using your knowledge and biases will only spoil it.

And remember, reading and interpreting the numbers is a skill that can be developed. So can discipline, holding off instant gratification and an ability to contain irrational exuberance.

Master these skills and you will find that with time, your abilities will reflect in the increasing accuracy and objectivity of the tests.

What have been the other common mistakes you have encountered in split testing? Let us know and we can start a discussion.