Thursday, April 21, was Emerce Conversion. Finally we could meet live again and learn from interesting speakers. This was the first time that Emerce Conversion did not take place in Pakhuis de Zwijger, but in the beautiful InnStyle in Maarssen. With speakers from VodafoneZiggo, Plus, ING and bol.com, among others, we got a good impression of what's going on in the experimentation landscape. 

The day began with an opening by Ton Wesseling. He took us through the history of Emerce Conversion and our field. Where in 2011 we were still talking about A/B test cases with high uplifts, this year it's much more about quality and strategy: doing good research and setting up an experimentation culture. 

Here's how to create a data- and validation-driven culture 

Mariska Buijs - ANWB

Mariska talked about the transition of CRO within ANWB. It started with a small team of analysts and now it is a standardized process in almost all business lines. The most important message: start validation-driven first and only then data-driven. To get an organization on board, it is important to first ensure that everyone knows about CRO and that adjustments are validated through experiments. This often involves testing features that have already been built or ideas from stakeholders. Start by validating these ideas first and include your stakeholders in the process and results. In case of a losing experiment, emphasize what the change would cost in revenue if it were implemented without an experiment. Grab as large a stage as possible to present your results.

Only when this happens structurally can you actually take a data-driven approach to this process and initiate the validations from the data. This data-driven approach ensures that the quality of the experiments goes up and thus your CRO program grows in maturity. The switch from a validation-driven to data-driven CRO program is not made overnight. Involve stakeholders in hypothesis sessions to introduce them to the data-driven way of working. Be more critical of new ideas: on what data is this idea based? Give inspiration sessions on how to do good research based on the 6V model and last but not least, document everything. 

Building up an experimentation mindset to elevate product outcomes

Luis Trindade - FARFETCH

Experimentation is a very important part of the success of online fashion platform FARFETCH. In fact, it is part of the DNA of the organization. During his presentation, Luis shared learnings and insights about creating an experimentation mindset in the organization. Using the Experimentation Growth Model, he showed how FARFETCH has become a Center of Excellence where decisions and strategies are driven by data and validations. 

He describes 3 pillars on which experimentation rests, namely:

1. Sharing and documenting knowledge: save all learnings and document the process in a central location.
2. Adoption and engagement: start small. You cannot change the culture in 1 day.
3. The right platform: choose the right experimentation platform for your organization. At FARFETCH, they built their own solution. 

Once these three pillars are in place, you can start working on your organization's experimentation mindset. Luis gave some tips for that:

• Get management on board;
• Think of experimentation as a mindset, not a tool;
• Continuously evaluate the maturity of the experimentation program;
• Experimentation is nowhere without a clear strategy. Use experimentation to fuel strategy. 

Experimenting at bol.com: from pioneering to platform

Denise Visser - bol.com 

During this session, Denise took us through the journey from 100 experiments in a year to winning a Experimentation Culture Award. 

Step 1: Start with a structured process.

Step 2: Scale up and share knowledge: “preach.” Involve as many people as possible.

Step 3: Standardize and document: make it very easy to create an experiment.

The challenge with step 3 is that this can come at the expense of the quality of experiments. For example, if anyone can set up an experiment, wrong hypotheses will be made. Try not to take on the role of policeman in this regard. And most importantly: persevere. Change management takes a long time. Use the power of repetition and remember that resistance is allowed. 

The power of long-term serverside A/B testing

Lucia van den Brink - Nu.nl

Lucia talked about the pros and cons of long-term server side testing. Server side testing becomes interesting when you have reached a certain level of maturity as an organization (see figure).

Server side testing has advantages and disadvantages. A major advantage is that there is no flickering effect present and winners can be implemented quickly. One of the disadvantages is that development must have capacity to build the tests in the backend. 

If you then get started with server side testing, you also have the ability to make tests run longer because the page is no longer cookie dependent. But is it wise to let a test run for a long time? That depends a lot on the visitor behavior on your website / in your app. And that has everything to do with habits. Do visitors come back often enough to build a habit? 

At nu.nl, visitors are true creatures of habit. This is because visitors return very regularly. A change can have a negative effect in the short term and a positive effect in the long term. This is why long-term experiments are regularly run at nu.nl to see the real effect on visitor behavior. Similar sites/apps where this is common are weather apps and banking sites. 

On to Emerce Conversion 2023

This 2022 edition of Emerce Conversion was a successful one. Put April 20, 2023 firmly in your calendar for the next edition!