Autopilot Mode is a feature designed to maximise the value of the better performing variation of your AB Tests. Normally, A/B testing consists of a period of fixed traffic allocation as below:
Once you have the results and know which is the better performing variant, you direct all of your traffic to it. However, depending on your traffic and significance level, an AB test can take weeks. In the above configuration, for the duration of the experiment you are diverting a large proportion of potential customers to a less optimal variant.
In statistics, this is known as the Exploration vs. Exploitation dilemma. In AB testing, the exploration phase involves testing variations to discover which results in the best conversion rate, whilst the exploitation phase is when you adopt the best performing variation and reap the rewards.
Before Autopilot Mode, your testing would move from a period of pure exploration to one of pure exploitation, meaning that during the test you have no opportunity to capitalise on the results. However, whilst initial results may indicate B as the better variant, you must still test A until you reach a big enough sample size to be confident of the results.
Autopilot Mode continually assesses the best performing variant in your test and allocates it a higher percentage of traffic. This smooths the process of optimization, and in doing so ensures you maximise conversions whilst running tests.
If in week 1 variation A is performing better, Autopilot would start gradually increasing the traffic to A. If in week 2 variation B started showing the better conversion rate, the traffic would gradually be reallocated back. By the time the experiment finishes in week 4, the winning variation will have had the majority of traffic for the 4 week test – and you won’t have had to change a thing!
On the results page of your experiment, you can follow how the best performing variation is gradually allocated more traffic than the Original.
The Autopilot redirects traffic gradually, not all at once. If a variation that has not been performing well, suddenly sees a great uplift, more traffic will be allocated to it gradually, but it won’t make a 180-degree turn in a day’s time.
Below, you can read exactly how frequently and how much traffic Autopilot reallocates.
The Autopilot reallocates traffic once a day, for every 1000 visitors that are tested in your experiment.
This means that Autopilot checks daily if 1000 visitors or more were tested since the last time the traffic was reallocated. If this is the case, it will go ahead and move traffic. If less than 1000 visitors have been tested since the last time traffic was reallocated, Autopilot will wait and check again the next day.
If the page you are testing has a lot of traffic (more than 1000 visitors per day), the Autopilot will reallocate traffic every day. If you have less than 1000 visitors per day, Autopilot will reallocate traffic less often.
Each time Autopilot reallocates traffic, it moves a percentage of traffic from the worse to the better performing page variations. The percentage depends on the number of page variations you created.
If you have 2 or 3 page variations (your original page + 1 or 2 Scenarios), Autopilot takes 7% of traffic the worst performing variation and gives it to the best performing variation. If you have 3 variations, the middle one will stay unaffected.
If you have 2 or 3 page variations (your original page + 3 or 4 Scenarios), Autopilot takes 5% of traffic the worst performing variation and gives it to the best performing variation. Then it takes 3% of the second worst performing variation and gives it to the second best performing variation. If you have 5 variations, the middle one will stay unaffected.
If you have 6 or more page variations (your original page + 5 or more Scenarios), Autopilot takes 3% of traffic the worst performing variation and gives it to the best performing variation. Then it takes 2% of the second worst performing variation and gives it to the second best performing variation. The other variations will stay unaffected.
Yes and no.
It is possible for the worst performing version to end up with 0% of traffic, being effectively paused. If, however, this worst performing version is the Original (Baseline), we anchor the lowest traffic allocation to 5% of traffic.
Yes, they can.
As described above, it is possible for one of the variations to be effectively paused by receiving 0% of the traffic. As Autopilot continues to check performance of the variations, it is possible that a previously high performing variation starts to perform badly.
If this situation happens, Autopilot will compare this performance with that of previously paused variations. If it is worse than any of the paused variations, traffic will be taken from this variation and allocated to the previously paused version, effectively re-activating it.