Lukas Vermeer, director of experimentation at Booking.com, presented during CH2019 on the trick for doing better experiments, namely sample ratio mismatch (SRM). 

In a sample ratio mismatch, the sample ratio does not match the design of an experiment. The distribution between control and variant is never exactly 50/50 and with an SRM check you check whether this difference is due to chance or not. Lukas addressed the questions during his presentation: why is it important to check for SRM? How do you know if there is an SRM? What causes it? And what can you do about it?

Why is it important to check for SRM?

Suppose that in the variant a specific segment does very well compared to control and that these visitors are much more frequent in the variant compared to control. That specific segment is then overrepresented and magnifies the positive effect. If the segment is underrepresented in the variant, the effect of the segment is actually reduced because it is underrepresented. SRM can thus lead to unreliable results. 

Lukas gave a good example of this in the paper: 'The pitfalls of experimenting on the web'. In it, they perform laughable experiments in which they could predict the outcome of the experiment. This is caused by a disproportionate distribution in a certain segment.

How do you know if there is an SRM?

SRM is everywhere! And it's one of the biggest problems in A/B testing and in science. Many people just don't check for SRM yet. This also became very clear when Lukas asked the audience who was doing SRM checks and only a few people raised their hand. There are on average between 6% and 10% SRM errors in tests (at Linkedin: 10%, Microsoft: 6% and at Booking even less), so enough reason to check.

But how do you find out if you have an SRM? For this, you can use the SRM checker of Lukas use. This SRM test can be used to identify data quality issues that may affect your A/B testing. 

Here, the boundary condition is a P value below 0.001. This means that it can be said with a certainty of 99.9% that the difference is not due to chance and that something went wrong.

What is the cause of SRM?

 Lukas gave the following diagram of possible causes of SRMs. 

What to do in the event of an SRM?

The first thing you want to do is find out where the SRM originated. Lukas uses the checklist below to do this.

After you have found the cause and fixed the problem, unfortunately there is nothing left but to restart your test as the current results are not reliable.

Want to know more about sample ratio mismatch?

Read more about SRM in Lukas' paper: 'Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners'. 

In addition, Lukas gave LinkedIn's paper 'Automatic Detection and Diagnosis of Biased Online Experiments.' and the paper 'Shining a light on dark patterns' as reading tips.

Short video of the presentation

Slides