Docs
When you launch an A/B test, you will collect data according to the change you made on each of the variations you have set up. Therefore, if you change your variation during the test, it will affect the results of your experiment.
Let’s imagine you’re running an experiment and it’s bringing a slight increase in conversion rate. Then you decide, while this experiment is still running, to change something else on the page you are testing. Your conversion rate then decreases slightly. It’s impossible to attribute the results to one of the two changes you made because both are running at the same time.
Our recommendation is to pause the experiment then create a new variation with the second change you want to apply. This way all the data from the first variation is preserved.