Many companies fear interaction effects
All companies that run online experiments come to a point where they want to run more experiments than they have important pages. An obvious solution is: run multiple A/B tests simultaneously on the same page. Yet most companies don't dare do this for fear of interaction effects.
In this blog, I describe three key benefits of running multiple experiments simultaneously on the same pages. I then address the risk of interaction effects. Do the benefits of running multiple A/B tests outweigh the risk of interaction effects? And can you overcome this risk?

A/B testing on high value pages yields the most results

A/B testing pays off the most when you run it on the most valuable pages of your website. On a page where all visitors pass by, you can quickly achieve a much higher return with a small improvement in design than on a page where only some of your visitors need to be. Moreover, on these pages the power is the highest, allowing you to detect smaller effects and giving you a shorter duration per A/B testing need. Logically, as a CRO, you spend most of your time and attention optimizing these pages.

Unfortunately, the number of high value pages on a Web site is limited. For most e-commerce websites, they can be counted on one hand: Product Listing Page (PLP), Product Detail Page (PDP), a checkout step or two, and maybe the home page. This is not a problem in the early stages of an optimization program. But what do you do when you get to the point where you have an A/B test live almost continuously on each of the aforementioned pages?

Benefits of running multiple A/B tests simultaneously on the same page

Running multiple experiments simultaneously on the same page has major advantages over other solutions that companies choose when they reach the point where continuous A/B testing is live on each of the high value pages.

• A/B testing on the high value pages yields more compared to testing on other pages. Therefore, even more testing on these pages will yield more than running tests on pages that, while not yet engaged, also represent less value;
• Power of testing remains high because you continue to use all traffic on the page for each test. This is not the case with the commonly used solutions to the space problem on key pages where traffic is split into different test lanes or multivariate testing (A/B/C/n) is done. The same visitor can then never be in multiple A/B tests at the same time, but this comes at the expense of the power of each test;
• Scheduling is easier because there is no need to consider pages that are occupied.

Does the risk of interaction effects outweigh these benefits?

One disadvantage of running multiple A/B tests simultaneously is that experiments can affect each other. The question is:

• How often does an interaction effect occur when we run two tests simultaneously;
• To what extent is it a problem when an interaction effect occurs;
• And should you or shouldn't you run multiple A/B tests on the same page?

How often does it occur?

To start with that first question, the experience of large organizations that do a lot of experiments such as Microsoft and Booking.com shows that interaction effects are rare (according to those responsible for the experimentation program at these companies, respectively Ronny Kohavi and Lukas Vermeer). Also our own experience at large online companies in the Netherlands is that interaction effects are rare, namely in less than 5% of the cases where multiple experiments have been live at the same time.

To what extent is it a problem when there is an interaction effect?

An interaction effect can be either beneficial or detrimental. It is advantageous when there is a positive interaction: the two variants reinforce each other and combine to provide an extra large uplift.

Figure 1: Visitors who saw variant B in both tests had a higher Conversion Rate % than other visitors
The interaction effect can also be negative: the variant in one test then weakens the effect of your other test so that a winning effect is no longer recognized.

Figure 2: Visitors who saw variant B in both tests had lower Conversion Rate % than other visitors

Should you or shouldn't you run multiple A/B tests on the same page?

Compared to the great benefits for value and power of your experiment program, the disadvantages of interaction effects are small. That's why we do recommend running multiple A/B tests on your most important pages.
Of course, it's a good idea to check whether there is an interaction effect at all. To make this easy for you, we have a Interaction Effect Calculator built. This can be found in the Online Dialogue toolkit.

Figure 3: The Interaction Effect Calculator from the Online Dialogue toolkit

... Just a quick note: we send out a newsletter every three weeks that includes the latest blogs, team updates and, of course, news about the offerings in our academy. Click here to subscribe.

Newsletter sign up

By entering the number of users and conversions for each of the possible variants, you quickly determine whether there is an interaction effect and, if so, whether it is a beneficial or detrimental effect.
This way, you overcome the risk of interaction effects and get the most out of the most important pages of your website!