A/B testing gets a ton of ink online.

But does it deserve it?

Is A/B testing the end-all, be-all for increasing conversions on your site or for product purchases?

The shower answer is no.

Here’s why, and what you should be doing instead.

The Problem(s) with A/B Testing

What, specifically, is considered a ‘good’ conversion rate?

Years ago, Larry Kim at Wordstream (an AdWords software company) set out to answer that very question, combing through over $3 billion in annual ad spend to determine where the conversion rate for most companies fell.

Here’s what he found:


(image source)

The vast majority of companies conversion rates were between the painfully low 0.5% and 2.5%

No compare that with the ones on the far right – the number of ones topping 11.5%!

Those were christened ‘unicorns’, because they were playing a different game entirely in order to drive above (above, above) average conversion rates that left competitors in the dust.

Before we dive into the three things they were doing, what weren’t they doing or wasting time on?

A/B testing minutia.

Although the internet is filled with triumphant stories where a simple headline change resulted in a huge conversion lift, that’s the rare exception and not the norm.

Most, in fact, fail to deliver any significant results. Many small results you do get will most likely “regress back to average” over time.

And in most cases, you probably don’t even have enough volume to make it worth considering in the first place.

Conversion expert Peep Laja from ConversionXL recommends that you wait to run any A/B tests until you have at least 1,000 monthly conversions and the ability to let a test run for at least 250 conversions before you can successfully (and confidently) declare it a winner.

So… what else?

If the ‘unicorns’ topping 10% aren’t running A/B tests (like most blog posts on the internet would have you believe), what are they doing?

Here are three places to start according to Larry Kim’s research.

Step 1. Start with Testing Your Offer

The most successful ‘unicorn’ companies all have one thing in common…

A “massively differentiated offer” that they’re promoting.

Most companies offer the same old boring free consultation or generic eBook to entice visitors to download. Just like everyone else. And unsurprisingly, they get results just like everyone else too.

One of my favorite examples from a non-obvious industry is the Lowe’s Personalized Lawn Care Plan:


It’s a tool, where you can go in and fill out a basic questionnaire that will eventually provide you with a tailored lawn care plan for the entire year, based on things like the climate where you live and even the specific kind of grass you have (did you know there was more than one?!).

Another example includes WordStream’s own Ad Grader which gets front-and-center billing on their homepage:


It’s another interactive tool that ‘grades’ your current performance and calls out areas for improvements along with a few helpful recommendations.

‘Good’ offers like these last two have a few things in common. Specifically, they need to be:

  1. Solve Painful (or Expensive) Problems: Let’s face it – most of us are driven by some amount of fear. We have pain points in our life (like those extra love handles) that we tend to obsess over. Good offers go straight to these ‘hot’ issues and promise an easy solution.
  2. Easy to Promote: Root canals are useful. But not popular. What’s difficult about creating an effective offer is that it also needs to appeal to a wide audience while solving painful problems at the same time. Teeth whitening on the other hand (appealing to vanity) might be a great idea instead.
  3. Utility: Why haven’t you downloaded yet another eBook lately? Because you can never find the time to read the thing! People are too busy to waste time on something that’s not going to offer them 10x the value when required to give over their personal information and spend the next ~10 minutes fiddling with your offer.

Step #2. Optimize Your ‘User Flow’

The second area ‘unicorns’ excel (that others neglect) is doing a better job of helping people find what they’re looking for, faster and easier.

What does that mean?

First, start by figuring out who people are and what they’re looking for based on where they come from. Specifically, the acquisition or traffic channel they use to reach you can tell you a lot about what other pages or messaging they should see next.

For example, someone typing in your URL directly into their address bar is probably looking for general company information and will browse around product or service pages. While those clicking on a PPC ad should instead bypass all of these pages and go directly to a perfectly-tailored landing page to purchase.


(image source)

After looking through your analytics to find all of the possible places that send you traffic, start looking at the ‘paths’ those people take through your site. For example, Google Analytics even has a free User Flow feature that can help.


The first place to start should probably be your ‘checkout’ funnel.

Where are user’s dropping off? Are there specific pages that act as a bottleneck? Are there specific questions that might confuse people or cause them to delay? (This also applies to service companies which might have longer-than-needed form fields.)

For example, one site shows the following pop-up before you can add the product to your cart:


Although there are perfectly logical reasons why this company would show this message, from a consumer’s standpoint, it’s kind of scary!

Full of strong language like Cautions! and Alerts!. Probably not the best message to send before you let someone add the product to the cart.

Instead, one way to improve this would be to simply relocate the entire thing towards the very end of your purchase process. By that point, someone’s already gone through the hassle of adding it to their cart, entering their billing and shipping details. This then becomes a Terms and Conditions fine print checkbox item and you’re good to go.

Step #3. Test MANY Different Pages

Most companies – if they are testing landing pages at all – are only doing a handful.

‘Unicorns’ on the other hand are busy cranking away, testing at least 10 in order to find that one awesome page.


(image source)

The reasoning? You’re not exactly sure what’s going to work. Maybe it’s a long, in-depth page vs. a shorter one. Maybe it’s several different value propositions, one positive and one negative.

The point, is that you’re testing larger assumptions – not single item variations like a classic A/B test of one button color vs. another.

While creating all of these page variations are undoubtedly time-consuming, the good news is that you can use Google Analytics (which is free) to run these conversion tests.

If you open it up, and click under the Behavior section, you should see Experiments towards the bottom.


Google’s ‘Content Experiments’ feature is perfectly suited for these types of tests, focusing more on the holistic performance of your pages (as opposed to single variables). It even allows you to run tests on 10 pages at once.


Here are the steps to get started:

  1. First up, decide on your Goal. Sometimes this is a hard goal (like opt-ins or purchases), while other times it’s a soft one (like Bounces or Time on Site).
  2. Next, drop in the URLs of the page you’d like to test, along with a few different variations. The catch here is that you’re obviously going to need to have all of these pages finished and live before this point in time.
  3. Next, comes the tricky part. You’ll need to properly set up your experiment code on your Original page variation. All you have to do is copy and paste a piece of javascript onto the code, but you might need some technical assistance.
  4. The fourth step is a mere formality. Their system will automatically check page variations for you to make sure the tracking code is installed properly and everything looks good to go.


A/B testing a single variable on your landing pages is great.

For large companies with tons of volume and a lot of resources to go around.

But for smaller companies that may not have the same luxuries, you’d be better off putting your limited resources into other areas.

Specifically, start by focusing on ways to improve the offer you’re using to generate new leads. Next, try to analyze how users are ‘flowing’ through your site and come up with ways to streamline it or make it faster and easier for them. Next, spend time creating many different landing pages to make sure you’re exhausting all options and increasing chances to find that one awesome page.

This process is still time-consuming.

But the good news is that it should also work more predictably too.