Suppose you run an A/B test where you show product recommendations on the homepage. At the end of the experiment, you find a winner: the variant with the recommendations has a significantly higher conversion rate. Great news, but does this modification also increase the average order value (AOV)? A calculation of the AOV per variant is insufficient here, because it does not tell you whether any difference is significant. In this blog, I explain what you can do so that you can immediately apply this in your A/B test analyses.

Average order value = continuous variables 

Most A/B testing calculators calculate a Bayesian probability or use a frequentist t-test to demonstrate significance of a measured effect. These calculators can handle variables where the outcome can have exactly two variants, such as conversion or no conversion. We call these binary variables. In the case of order value, we talk about a continuous variable. This is because the possible outcomes have a wide spread, and can range from a purchase of less than 10 euros to large orders of hundreds of euros. For this type of variable, most of the previously mentioned calculators do not work. Fortunately, there is an alternative: the Mann-Whitney U test.

... Just a quick note: we send out a newsletter every three weeks that includes the latest blogs, team updates and, of course, news about the offerings in our academy. Click here to subscribe.

Newsletter sign up

The Mann-Whitney U test explained

The Mann-Whitney U key is a statistical test that tells you whether a variable is significantly different between two groups. In addition to being able to deal with continuous data (such as mean order value), this test has another advantage: the test is nonparametric. This means that no assumptions are made about the underlying distribution of the data. This is useful when, for example, the data is not normally distributed (see Figure 1). And this is convenient, because this is usually the case with variables relevant to A/B testing, such as AOV and time on page. Namely, this data has a lower bound at zero, while the maximum is unbounded. This causes the distribution to have a shape similar to the image below.

Apply in practice

Getting started with the Mann-Whitney U test yourself, but no idea how to get started? We've added a Continuous Metric Calculator to our CRO toolkit! After you create a free account, you can get started. You fill in here yourself the results found in an A/B test, and the tool calculates whether there is a significant difference between the two variants. In this way, you make continuous variables part of your A/B test analysis as well.

Have questions about using this tool? Email them to: analisten@onlinedialogue.com

You might also find this interesting: