Make an appointment

Twinkle Conversion Rubric November 2018: Separate or simultaneous A/B testing

Reinier Koolmees

Reinier Koolmees

Data Analyst

20-11-2018 - minuten leestijd

This article was written by Anouk Erens, senior data insights expert at Online Dialogue, for the online analysis and optimization platform Webanalisten.nl. The complete article can be read at https://anali.st/interactieeffecten.

You're enjoying A/B testing and you want to increase the number of experiments, but you've reached your limit. Do you then start experimenting on multiple pages at the same time? Does this help in speeding up your conversion rate optimization program? Or are you just polluting your data? Unfortunately, the answer to both of the latter questions is yes. Do the advantages outweigh the disadvantages?

The advantage

Suppose you have four pages on which to experiment, and on all four pages an experiment should be live for four weeks. If you wait each time until one experiment is finished before putting a new one live, and thus you only test one page at a time, you can complete twelve A/B tests a year. However, if you put multiple experiments live at the same time, four experiments per month can be live at the same time. That's 48 experiments on an annual basis, and so you're four times more likely to learn something and less likely to lose money.

The downside

The consequence, however, is that you run a greater risk of interaction effects, where the effect of one test affects the effect of another. Two scenarios are possible here.

Scenario 1

When there are no interaction effects, you can assume that it is fine to have multiple tests live at the same time without polluting your data. Besides, your competitor is probably also experimenting, and their A/B tests will have as much impact on your A/B test results as your own experiments.

Scenario 2

When an interaction effect does occur, it can lead to:

  • A positive effect: that is, the effect of variant B of experiment 1 amplifies the effect of variant B of experiment 2. This makes both tests a winner, and so you implement both tests. However, this effect is only positive when you have both variants live. If you do not do this and implement only one variant, the variant may produce a completely different effect.

simultaneous A/B testing

  • A negative effect: that is, variant B of experiment 1 dilutes a possible positive effect of variant B of experiment 2. As a result, both A/B tests do not seem to produce winning variants, resulting in a drop in your win-to-ratio, even though there may be positive effects.

simultaneous A/B testing

Make the trade-off

Unfortunately, we don't know in advance whether there are interaction effects, so it's impossible to give an unequivocal answer as to whether multiple A/B tests at the same time are a problem. This makes it all the more important to weigh the pros and cons. If you expect large overlap, or if your experiment is very important: test separately. If not, go for speed and test simultaneously.

Click image to enlarge, or download here the PDF.

Simultaneous A/B testing

Reinier Koolmees

Reinier Koolmees

Data Analyst

On a daily basis, Reinier is an analyst at Online Dialogue working on data-driven conversion optimization and translating data into insights. Online Dialogue has been a leading CRO agency for 10 years, where people, quality and knowledge transfer are key. We help our clients make better and more reliable decisions using data and psychology.

Deze blogs vind je vast ook interessant
Secret Link