This article was written by Anouk Erens, senior data insights expert at Online Dialogue, for the online analytics and optimization platform Webanalisten.nl. The complete article can be read at https://anali.st/datavalkuilenbijabtesten.

We all see them come by, at work, on LinkedIn, at conferences: examples of A/B testing with 10 percent, 20 percent and 25 percent uplift. Nice to see what other companies are doing and good inspiration to enrich your own optimization program with. But there are data pitfalls that can mislead you with these examples. Therefore, always ask the right questions.

1. What is the most important metric?

One of the first things I wonder about such A/B test examples is what metric was used to measure the result against. Ideally, you would like to bill an A/B test based on the actual revenue generated. But this is not always possible due to the number of visitors. Therefore, always tell what metric was chosen. After all, a 15 percent uplift in click-throughs is very different from the same uplift in orders and sales.

2. Are the numbers high enough?

We still see too often that experiments are carried out on pages on which there is actually far too little traffic to carry out reliable A/B tests. Therefore, whenever possible, always ask for the absolute numbers. Of course, an increase from 25 to 30 orders is very different from an increase from 600 to 720 orders. Still, both are a 20 percent uplift. For the same reason, it is important to check in advance how many visitors and conversions you need in order to test at all on a given page and whether the effect you see is actually statistically significant and not a coincidence. For this, use an A/B test significance calcula- tor such as Abtestguide.com.

3. What is the quality of the technical implementation?

Whether you have been doing A/B testing for a while or are just starting out, it is always important to properly check the imple- mentation of your testing tool. Don't blindly trust the numbers your A/B testing tool shows, but regularly check your data, including in your web analytics tool, to see if they are the same. Since you do the rest of your web analytics in this tool as well, it's nice to start from one truth.

4. What did your experiment yield in euros?

Finally, you want to check what your experiment generated in euros, but also what the costs were. Suppose the uplift earns you 20,000 euros in six months, but the development and ancillary costs are 17,500 euros, is it worth having your experiment built?

Click image to enlarge, or download here the PDF.