June 8, ODHouse was all about the 24th edition of Dialogue Thursday: Statistics & Algorithms. The program included presentations by Lucas Vermeer of Booking.com and Guido Jansen of Euroflorist. The day was introduced by Ton Wesseling and concluded with a panel discussion where the audience could ask questions of all three speakers.

Lukas Vermeer | Explaining statistical concepts through compelling stories and concrete examples

Almost everyone has had some exposure to statistics during their studies. But have you applied this knowledge throughout your career? If you understand statistics well, you will be better able to understand the results of your online experiments. This is very important since the results of your old experiments are also the input for new experiments. During his presentation, Lucas shared four statistical stories that helped scale Booking.com's testing culture. We wrote down two of them for you. Do you recognize the statistical problems?

Story #1: The Sharpshooter

The first story is about a sniper. And not just any sniper, as he uses a shotgun as a weapon (not the best choice if you need to shoot very precisely from afar). In the distance is a barn where the sniper fires a few shots. When he is done, he walks to the barn and draws a few red and white circles around each hole. This way it looks like each shot was a bull's-eye.

A similar approach is also frequently done in experiments, but it is completely useless. Running experiments only adds value if you can make a prediction prior to your test. Draw your target before you start shooting.

Story #2: Thinking about eyeliner causes weight loss

The reports of online survey tools often show how many people completed the survey, but not how many people started the survey. A simple experiment shows how the absence of this information can lead to unreliable results. Imagine an experiment involving two surveys. In survey A, participants must first imagine themselves putting on shaving cream and then answer a series of questions about age, sex and weight. In survey B, participants must imagine they are putting on eyeliner before answering the same questions. This experiment has a miraculous outcome: thinking about eyeliner causes weight loss.

Nothing could be further from the truth. Both men and women understand how shaving cream works and can therefore imagine it well. But men in general have more difficulty imagining someone putting on eyeliner which makes them more likely to leave the survey. This ensures that by the time participants get to the question about weight, only women are left. So always keep an eye on who has seen the experiment. Do many people drop out of one group and fewer in the other? The goal is to always keep the distribution 50/50.

Guido Jansen | How Euroflorist is preparing for Artificial Intelligence: changes to our optimization strategies

How do you get the best converting layout of your website? During his presentation, Guido showed how the current optimization process at Euroflorist is structured and how Euroflorist is preparing for the next steps for optimizing the Euroflorist B2C e-commerce platform through Artificial Intelligence systems. Thanks to Euroflorist's optimization team, 10 to 20 tests are running simultaneously and one in five tests provides uplift. But the team wants more.

 AI is the answer

After testing various AI applications, Euroflorist, with the support of Online Dialogue, got to work with the American-made tool Sentient. A tool for multivariate testing that evolves as the experiment progresses until the most optimal variation remains. Thanks to this tool, it is possible to test multiple elements of a Web site simultaneously and, without needing huge amounts of visitors, look for the optimal solution for your site.

We have a winner!

The first tests were run in 3 of the 11 countries in which Euroflorist operates: the Netherlands, Norway and Sweden. These countries were chosen as they differ a lot in visitor numbers and conversion rates. For each country, the same 8 elements on the site were tested with 2 variants each. The test has run up to 3 generations and there are already some results. The tests for the Netherlands and Norway did not prove to be significant, but Sweden on the other hand proved to have a winner. In the next phase, Euroflorist's optimization team is going to A/B test Sweden's winning variant to be sure of the effect.

What did we learn?

Looking back on the experiment, Guido listed the pros and cons of testing with Artificial Intelligence.

Benefits of testing with AI:

• The time per variant is much shorter
• You end up with an uncommon variation
• You need less traffic (about 75% less)
• Higher ROI

Disadvantages of testing with AI:

• Extended startup process
• It remains a black box
• You don't know how long it will be before your experiment passes to the next generation
• Garbage in, Garbage out
• It greatly frightens their local Ecom managers

Benefits normal A/B and MvT testing:

• More control
• More rigorous hypothesis testing

Disadvantages of normal A/B and MvT testing

• Slow
• Fewer variants
• More traffic required
• Limited test capacity

The slides from Guido's presentation can be found here: http://ondi.me/slidesdido24

September 28, 2017 marks the 25th edition of the Dialogue Thursday! Would you like to be there? Then sign up for the waiting list by filling out the form below:

[ninja_forms id=35]