Autopilot Mode: Put your AB Tests on Autopilot

What is Autopilot Mode?

Autopilot Mode is a feature designed to maximise the value of the better performing variation of your AB Tests. Normally, A/B testing consists of a period of fixed traffic allocation as below:

Once you have the results and know which is the better performing variant, you direct all of your traffic to it. However, depending on your traffic and significance level, an AB test can take weeks. In the above configuration, for the duration of the experiment you are diverting a large proportion of potential customers to a less optimal variant.

Explore vs Exploit

In statistics, this is known as the Exploration vs. Exploitation dilemma. In AB testing, the exploration phase involves testing variations to discover which results in the best conversion rate, whilst the exploitation phase is when you adopt the best performing variation and reap the rewards.

Before Autopilot Mode, your testing would move from a period of pure exploration to one of pure exploitation, meaning that during the test you have no opportunity to capitalise on the results. However, whilst initial results may indicate B as the better variant, you must still test A until you reach a big enough sample size to be confident of the results.

Let Autopilot Mode monitor results for you

Autopilot Mode continually assesses the best performing variant in your test and allocates it a higher percentage of traffic. This smooths the process of optimization, and in doing so ensures you maximise conversions whilst running tests.

If in week 1 variation A is performing better, Autopilot would start gradually increasing the traffic to A. If in week 2 variation B started showing the better conversion rate, the traffic would gradually be reallocated back. By the time the experiment finishes in week 4, the winning variation will have had the majority of traffic for the 4 week test – and you won’t have had to change a thing!

On the results page of your experiment, you can follow how the best performing variation is gradually allocated more traffic than the Original.