You're enjoying A/B testing but (because you're hitting your limit) you want to experiment more. The solution may be in testing on multiple pages simultaneously. But does this help in accelerating your CRO program? Or are you just polluting your data? Unfortunately, the answer to both is “yes.” The only question is whether the advantages outweigh the disadvantages.

Imagine the following scenario: a user enters a product page where he is in the control group of experiment 1. He clicks through to the shopping cart and sees variant B of experiment 2. He then fills in his details and sees variant B of experiment 3 and finally converts.

In such a case, it is difficult to determine which combination of variants provided a winner.

The benefit of multiple A/B tests in a user-journey

First, if you run multiple A/B tests in the same period, you can obviously put many more experiments live than if you don't. Say you have four pages in your funnel where you can experiment, and on all four pages an experiment should be live for four weeks. If you put multiple experiments live at the same time, four experiments per month can be live at the same time. This is 48 experiments on an annual basis. If you always wait until one experiment is finished before putting a new experiment live, and thus you only test one page at a time, this is 12 experiments per year.

Simultaneous testing of different pages in this case will accelerate your CRO program. As a result, you are four times more likely to learn something and reduce the chances of missing out on money.

The downside of multiple A/B tests in a user-journey

The most commonly heard disadvantage is the contamination of your data. This has to do with interaction effects. With an interaction effect, you assume that the effect of one test affects the effect of the other test. There are two possible scenarios here.

Scenario 1

In reality, there are no interaction effects, so the A/B tests do not affect each other. We can then assume that it is fine to have multiple tests live at the same time, without polluting your data. Besides, your competition is probably also experimenting and these experiments have as much influence on your experiment as your own experiments. So it may happen that a user is comparing different websites and also sees a competitor's experiment in several tabs.

Scenario 2

However, there is an interaction effect. This effect can lead to:

1. There is an interaction effect and the different tests have a positive effect on each other. That is, the effect of variant B of experiment 1 reinforces the effect of variant B of experiment 2. This makes both tests a winner, and so you implement both tests. However, this effect is only positive when you have both variants live. If you do not do this and implement only one variant, the variant may produce a completely different effect.
   
2. There is an interaction effect and the different tests have a negative effect on each other. That is, variant B of experiment 1 weakens a possible positive effect of variant B of experiment 2. As a result, both A/B tests do not seem to produce winning variants. As a result, your winner ratio drops, even though there may be positive effects.

Make the trade-off

Unfortunately, we do not know in advance whether interaction effects exist so there is no straightforward answer as to whether multiple A/B tests at the same time are a problem. But the more important it is to make a good consideration. If you expect large overlap (for example, responding to the same visitor need in both A/B tests) or your experiment is very important, test separately. If not, go for speed and test simultaneously.

The consideration you make is different every time, but that is precisely why it is important to think about it carefully. I am very curious about your experiences. What is your consideration of whether or not to experiment simultaneously within the same user-journey?

This article was published on Sept. 6 at Webanalists.com