KPN became an A/B testing factory: four tests a week for maximum learning speed.
What began as external execution is now an internally supported testing culture with its own teams.
All knowledge gained is recorded in a CRO manual that helps new employees get up to speed quickly.
In March 2018, I started as a UX designer on one of Online Dialogue's biggest projects. KPN wanted help with the data-driven optimization of their webshop environment. This would involve not one, not two, but four A/B testing per week startup. Together with a multidisciplinary team from Online Dialogue with analysts, psychologists, UX designers and a strategist, we were going to run four A/B tests per week for six months.
This was the start of a long and valuable collaboration, with great results and many learnings as a result. I would like to share the process developments we have experienced together with KPN over the past two years and how they contribute to the current cooperation.
For the first project at KPN, Online Dialogue was deployed as an ‘A/B test factory. This meant that we were hired as a full team to perform all A/B tests for KPN. By performing four A/B tests per week we would learn a lot about the online behavior of KPN users in a short period of time and increase the conversion rate. And with success: we achieved great results with the first A/B testing project. KPN was enthusiastic about what we had put in place.
After these first successful months, the focus shifted from ‘factory’ to ‘collaboration. From then on, handovers were initiated, with more and more roles hooked up from within KPN. For example, the experiments were picked up by internal developers from the beginning of 2019. In addition, analysts were also trained to take over the entire process. Both with success! The factory of seven people was converted to a 50/50 collaborative process. In the period that followed, the role of UX was also transferred to KPN so that the CRO team now consists largely of internal people from KPN. This was and is also one of the goals: to help organizations set up a CRO culture.
Online Dialogue is additionally still hooked up as a center of excellence. This means that we support different disciplines as well as different departments within KPN. The goal is to use the same working method everywhere within KPN. All the knowledge we have gained over the past two years has been summarized and compiled in the CRO manual. In this way the knowledge is guaranteed and new KPN employees are trained in the CRO team.
The A/B testing factory had many advantages, but also some disadvantages. As a team, we were completely attuned to each other, which resulted in the successful setting up of many experiments. However, because we were present in the department with a large group, this was experienced as a kind of ‘invasion. This led, logically, to some resistance in the beginning.
When the value of the A/B testing and the associated behavioral insights were confirmed after the initial process, the switch was made. Several people from KPN were hooked up and the department was actively informed of the process from then on. Weekly overview mails, open chairs at meetings and monthly lunch & learn sessions now result in increasing enthusiasm for experimentation among the KPN team. Everyone in the department has the opportunity to send in their own ideas, which are then prioritized within the CRO team. Nowadays, the number of people wanting to attend the weekly CRO meeting is so large that multiple teams have been created.
One of the biggest changes in A/B testing was the transition in development. In 2018, development still sat externally and all communication went through Online Dialogue. The exceptions on the website were not equally familiar to all developers, which caused extensive checks before going live with the experiment. In addition, mutual communication took extra time. As of January 2019, this changed. With the arrival of internally based developers, all code for A/B testing was built internally. Because the development team was present at KPN throughout the week, experiments could be quickly tuned and go live. Especially in periods where multiple A/B tests went live in a short period of time, this was a nice outcome. Development is an essential part of the A/B testing process and can ensure a lot of speed in the process if it is well organized and the right amount of time is set aside.
We experience it often with clients: redesigns. Every so often there is a desire to give the website a new look. This was also the case with KPN. A new proposition was introduced, which required a different look. Earlier experiments largely served as input for this redesign, but there were also new components that had not previously been validated. This gave reason to collect new data from the website and use it to set up new experiments.
Can you still use the ‘earlier’ insights in this? In part, you certainly can. Because the optimization program is set up in main and sub-hypotheses, we not only have learnings per test but also at the meta-level. These learnings come from multiple experiments and are therefore independent of design principles and can therefore be applied to the new design.
The experiment train we started two years ago has now developed into a well-oiled machine within KPN. We are seeing more and more enthusiasm from the online department for data and psychology driven optimization. With the most important achievement being that experimentation is becoming a regular part of the digital department and eventually an increasing part of the work is being validated.
Want to learn more about our knowledge and experience setting up an evidence-based optimization program for your organization? We are happy to help you!
Nederlands Nederlands
English