There’s a lot of controversy around one-tailed vs two-tailed testing.
Articles like this lambast the shortcomings of one-tailed testing, saying that “unsophisticated users love them.” On the flip side, some articles and discussions take a more balanced approach and say there’s a time and a place for both.
In fact, many people don’t realize that there are two ways to determine whether an experiment’s results are statistically valid. There’s still a lot of confusion and misunderstanding about one-tailed and two-tailed testing.
The commotion comes from a justifiable worry: are my lifts imaginary? As mentioned in this SumAll article, sometimes A/A tests will come up with some quirky results, thus making you question the efficacy of your tools and your a/b testing plan. So when we’re talking about 1-tailed vs 2-tailed tests, we’re really talking about whether or not we can trust the results of our a/b tests and take action based on them.
What’s the difference? Does it matter? When should I use one-tailed? Two-tailed?
One-tailed vs two-tailed – what’s the difference?
If you’re just learning about testing, Khan Academy offers a clearly laid out illustration of the difference between one-tailed and two-tailed tests:
In essence, one-tailed tests allow for the possibility of an effect in just one direction where with two-tailed tests, you are testing for the possibility of an effect in two directions – both positive and negative.
Chris Stucchio does a great job explaining the difference between the two tests in context:
“In frequentist tests, you have a null hypothesis. The null hypothesis is what you believe to be true absent evidence to the contrary. Now suppose you’ve run a test and received a p-value. The p-value represents the probability of seeing a result at least that “extreme” in the event the null hypothesis were true. The lower the p-value, the less plausible it is that the null hypothesis is true.Now suppose you are A/B testing a control and a variation, and you want to measure the difference in conversion rate between both variants.The two tailed test takes as a null hypothesis the belief that both variations have equal conversion rates.
The one tailed test takes as a null hypothesis the belief that the variation is not better than the control, but could be worse.” (quote source)
Put simply, the two-tailed test can show evidence that the control and variation are different, but the one-tailed test is used to show evidence if variation is better than the control.
Does it matter which method one you use?
Okay, so now that we went over what the tests actually are, we can ask the important question: does it even matter which you use? Turns out, that’s a complicated question. It’s where a lot of the derision arises.
According to Kyle Rush, Director of Frontend Engineering and Optimization at Hillary for America, it does:
“The benefit to using a one-tailed test is that it require fewer subjects to reach significance. A two-tailed test splits your significance level and applies it in both directions, thus each direction is only half as strong as a one-tailed test (which puts all the significance in one direction) and thus requires more subjects to reach significance.
Pros and Cons of Each Method
Maxymiser laid out some pros and cons of using either test (they also made a handy little infographic if you’re having trouble understanding the difference):
Pros Cons One-Tailed Tests
- Requires less traffic
- Gains significance faster (read: why significance does not equal validity)
- Only accounts for one scenario
- Can lead to inaccurate and biased results
- Accounts for all three scenarios
- Leads to accurate and reliable results
- Requires more traffic
- Takes longer to gain significance
Other factors in validity
According to Andrew Anderson from Malwarebytes, though, there are many other factors at play when testing:
“You also have the issue of a T-Test versus a Z-Test or other Gaussian frequentist approaches versus a Bayesian approach. It is really not as simple as one-tailed versus two-tailed. All of this also assumes that you are testing in an environment where confidence is even slightly useful and that are validating the underlying assumptions behind the model. For example, in my last two positions, variance has been so high that I don’t even look at confidence as it in no way reflects real world situations.”
So there are other factors when it comes to testing for statistical validity. Still, there are strong opinions around one-tailed and two-tailed testing.
The case for two-tailed testing
Two-tailed tests mitigate type I errors (false positives) and cognitive bias errors. Furthermore, as Kyle Rush said, “unless you have a superb understanding of statistics, you should use a two-tailed test.”
Here’s what Andrew Anderson had to say:
“If given the option, you are much better off using two-tailed versus one-tailed as fundamentally one-tailed tests enable more type I errors and also cognitive bias errors. One-tail is tempting because it leads to faster “conclusions” but that just magnifies all the problems with using confidence and does nothing to add value to your organization. Both are based on similar model assumptions, things like no variance (only population error rate), unbiased sampling, and Gaussian distribution, so they are both very rarely reflective of real world situations, but the higher bar of difficulty to reach “significance” in a two-tailed test, as well as the measure of both directions better reflects real world outcomes as most of the time what we think will be positive is actually negative (see 10% industry average success rate), as well as mitigates (barely) the risk of acting too quickly.”
Neal Cole, Conversion Specialist at a leading online gaming company, agrees:
“Personally I think one tail tests are not appropriate for most A/B testing and increase the risk of a false positive. When we are A/B testing we can’t be sure of the direction of the difference in the key metrics. Around 50% of tests don’t deliver an uplift in conversion and so we are kidding ourselves if we use a one-tail test as our default approach.”
When can I use one-tailed tests?
According to some, there is a time and a place for a/b testing. It is often contextual and depends on how you intend to act on the data. As Luke Stokebrand said, “One-tailed tests are not always bad, it is just important to understand their downside. In fact, there are many times when it makes sense to use an one-tailed test to validate your data.”
Andy Hunt from UpliftROI, though acknowledging the faults of one-tailed tests, takes a realistic approach:
“Two-sided tests are better but not the norm for most marketers or VWO/Optimizely.
Instead of focusing on whether to use two or one sided tests, you’d be better off simply letting the tests run for more time to be sure the results are valid AND running “A/A” tests where you make sure the control and variation are performing exactly the same.”
Similarly, Jeff Sauro from MeasuringU reiterates that while you should normally use the 2-sided p-value, “you should only use the 1-sided p-value when you have a very strong reason to suspect that one version is really superior to the other.”
Kyle Rush echoes this:
“If you are testing a new variation on your website and only want to know if the variation is better, then you would use a one-tailed test in that direction. If you don’t plan on deploying the variation if it doesn’t win, then this is a completely fine approach. However, this turns into a bad approach if you deploy the variation when it wasn’t a statistically significant winner because the one directional test didn’t measure the hypothesis in the other direction, so you don’t know if the variation is worse. A two-tailed test applies the hypothesis test in both directions so this issue is mitigated.”
Which tools use which method?
When you ask the question of which a/b testing software uses which method, you enter a world of murky answers and ambiguity. That’s to say, not many of them list it specifically. So here’s what I got from research and from asking testing experts (correct me if I’m wrong or need to add something):
Tools that use one-tailed tests
- Optimizely (two one-tail tests)
- Google Content Experiments (uses Bandit, but one-tail if you disable that feature)
Tools that use two-tailed tests:
- Adobe Target
Of course, certain tools have custom frameworks as well (like Google’s Multi-armed Bandit). Kyle Rush explains Optimizely’s Stats Engine:
“In terms of tools on the market, Optimizely’s Stats Engine makes stats and questions like this very easy for you, much more than any other platform out there. A huge benefit to Stats Engine is that with traditional one and two tailed t-tests you have to calculate a sample size based on an arbitrary variable called minimum detectable effect (MDE). With Stats Engine, you don’t have to calculate a sample size, i.e. picking an arbitrary MDE. This has a huge impact because you will often underestimate your MDE when calculating sample size (because you’re just picking the number out of thin air) which has the effect of requiring a sample size that can be exponentially larger than what you actually needed to reach significance.
I strongly believe that worrying about questions like this is a thing of the past with Optimizely’s Stats Engine.”
The issue of using 1-tailed vs 2-tailed testing is important, though the decision can’t be made with statistics alone. As Chris Stucchio said, “it needs to be decided from within the context of a decision procedure.”
He continues to say that, “When running an A/B test, the goal almost always to increase conversions rather than simply answer idle curiosity. To decide whether one tailed or two tailed is right for you, you need to understand your entire decision procedure rather than simply the statistics.”
So if you’d like to learn more about one-tailed and two-tailed testing, there are many resources. Here are a few that are easy to understand:
Otherwise, I’ll close with something Peep said about the subject: “One vs two-tailed issue is minor (it’s perfectly fine to use a one tailed test in a lot of cases) compared to test sample sizes and test duration. Ending tests too soon is by far #1 testing sin there is.”