To get reliable results from your A/B test, you need statistically significant data. Gathering enough data to produce statistical insights is the main reason why most tests run for longer than expected. AB test significance is achieved when you believe your results are NOT the result of random chance. So, a test that has achieved statistical significance at a 95% CONFIDENCE LEVEL would occur through chance 5% of the time (1 in every 20 similar experiments).
To calculate the sample size required for your test, you need to know the CONVERSION RATE for your original page and the EFFECT SIZE of your new scenario. You also need to set a confidence level that will be used to determine if your results are significant.
Your effect size is the magnitude of the difference produced by your new scenario. In website optimization this is usually your conversion rate UPLIFT (the percentage increase in your conversion rate).
With a higher original conversion rate, your effect size (the uplift caused by your experiment) will be larger. An uplift of 25% is much greater if your original conversion rate is 16% (Scenario 1) than it is if your original conversion rate is 4% (Scenario 2). Your new conversion rates would be…
If you knew in advance what the effect size of your experiment would be, you could calculate the sample size required to achieve AB test significance. However, in most cases, the effect size is unknown until midway through the test. That’s why Convertize predicts the days required your experiment to achieve significance at the same time as calculating the effect size of your test.
These are some A/B testing scenarios, based on common parameters…
For a step-by-step guide to AB test significance and sample sizes, read this Introduction to A/B testing.
If you have performed an A/B test and you want to calculate the significance of the results yourself, you can use this simple AB testing calculator. All you need to know is the number of visitors your pages have received and the number of conversions they have produced. The calculator assumes a confidence level of 95% (as this is the industry standard).
To find out more about statistical theory and website optimization, read this article on A/B testing sample size.
The benefit of using Convertize is that you can apply optimization principles without needing vast amounts of traffic. Convertize’s optimization library is based on scientific research and proven UX principles. Of course, it is better to test them before making your changes final, because the impact of your tests will depend on their context.