Dutch Railways (NS) transports millions of passengers every day and is a crucial link in the mobility of the Netherlands. How does an organization of this size develop a successful experimentation and validation culture? We talked about this with Jeroen Oosting, responsible for the CRO program at NS. "Every day I get the chance to contribute to a greener future for the Netherlands." said Jeroen. Together with Online Dialogue, he worked on a strategy to embed CRO in daily processes.

In this interview, Jeroen talks about the challenges, results and process of making CRO scalable within NS. From a pilot program to over 145 tests in one year. Read how NS not only built a successful CRO strategy thanks to collaboration with Online Dialogue, but also the prestigious Experimentation Culture Rising Star award won.

How do you scale CRO within a large organization like NS?

NS' internal organization; such as product owners, cluster leads, teams and clusters, among others, saw a lot of growth potential in Conversion Rate Optimization (CRO), but struggled with some challenges to build a successful CRO strategy:

• Lack of knowledge and capacity: both Product Owners (POs) and cluster leads lacked the time and expertise to successfully execute CRO.
• Support within the organization: convincing teams to deploy CRO on a large scale was a challenge, despite the positive results of an earlier pilot.
• Insufficient experiments: the number of experiments conducted was therefore limited.

The challenge was also in the political dynamics within clusters: teams often worked output-driven (focus on delivering features) rather than outcome-driven (focus on measurable results), which hindered a larger rollout of CRO. There were also challenges in creating support among product owners and cluster leads, who raised the alarm that time and knowledge were lacking for CRO initiatives.

The approach: collaboration with Online Dialogue

After a successful pilot, NS decided to scale up CRO further. Hiring additional full-time employees was not an option, so the desire arose for a training program for NS employees in each of the various clusters. Online Dialogue was selected as a partner because of its expertise in data analysis and psychology and accessible method of operation. "I chose mainly on person; I had a good click. Online Dialogue came across as flexible and looser, a culture that suits NS well. That personal click was ultimately the deciding factor." explains Jeroen Oosting.

For 14 months, Online Dialogue trained and coached 13 NS employees, allowing them to devote 25% of their time to CRO. The approach included:

• Kickstart the CRO program: specialists within NS, often new to the field, were trained to incorporate experimentation and validation into their work.
• Hands-on training and coaching: specialists learned how to set up experiments from A to Z, from formulating test ideas to presenting learnings. Extra attention was paid to skills such as data analysis, psychology, programming and presenting results.
• Safe learning environment: in bi-weekly multidisciplinary meetups, the teams were able to learn from each other, share successes and discuss insights in a safe and stimulating setting.
• Technical Support: team got help setting up the experiments, measurements, and automating analysis and reporting.

"The consulting role that Online Dialogue performed in addition to the trainings really confirmed their added value as a sparring partner," said Jeroen. This collaboration laid a solid foundation for a data-driven way of working within NS.

Outcome: from pilot to organization-wide adoption

Online Dialogue's approach of working with NS to train specialists within different clusters has paid off. After a year of collaboration, the results are telling and NS has built a successful CRO strategy in collaboration with Online Dialogue.

• Exponential growth in testing: the number of experiments increased from 27 (conducted by a central CRO team) to 145 tests spread throughout the organization.
• Cluster-specific successes: The Loose Ticket (LVB) cluster saw an increase of 3-4 times more tests compared to the previous year. This success was a major driver for further adoption of CRO within NS.

In the Loose Ticket (LVB) cluster, it was proven that CRO can lead to concrete business results. At the same time, some clusters lagged behind in the number of experiments, partly due to a focus on output over measurability. Work is now underway here to further integrate a culture of experimentation and validation to increase the impact of CRO.

A key insight from the collaboration was that not all expected results emerged from testing. It was precisely these unexpected outcomes that highlighted the importance of testing and validated the usefulness of a data-driven approach.

Lessons learned: the building blocks for a successful culture of experimentation

Working with Online Dialogue has provided NS with smart tips and insights to take experimentation and validation to the next level:

1. Internal knowledge building is crucial: training its own employees increases expertise within the organization and makes it future-proof.
2. Coaching in practice works: hands-on guidance provides immediate impact and accelerates CRO adoption.
3. Flexibility and commitment: informal, accessible cooperation promotes quick results and pleasant collaboration. Work = play!

Online Dialogue's training programs have ensured that knowledge about CRO is further secured within the organization. Moreover, a structural process has been established to train new employees and embed a data-driven way of working.

Outlook: an organization-wide validation culture

NS looks confidently to the future and is taking steps to further strengthen and embed CRO in the organization. The focus is on validating features with concrete business objectives, expanding A/B testing to clusters where experimentation is still in its infancy, and improving test quality by emphasizing impact over quantity. At the same time, NS continues to build a strong culture of experimentation by increasing support within clusters.

The Rising Star award, which NS received from the international professional jury of Experimentation Culture received, acknowledges the design and initial results of the CRO program.

With the help of Online Dialogue, NS has laid a strong foundation to increasingly integrate experimentation and validation into their organization, with room for further development and growth towards an organization-wide experimentation culture.

Online Dialogue on partnering with NS: making an impact on mobility and sustainability

At Online Dialogue, we are extremely proud to have been able to guide NS through this development process. It was a project from which we ourselves got an enormous amount of energy and satisfaction.

Ruben de Boer, Lead Conversion Manager at Online Dialogue: "The fine collaboration, from training to independent experimentation, the different challenges per team and conversations with stakeholders formed an exciting case in which we managed to make very nice steps together. In addition, NS' specialists were right at home at the Online Dialogue office, where we created a safe and inspiring environment to learn and grow. The happiness of NS about the cooperation and winning the Rising Star Experimentation Culture Award is a wonderful confirmation of this."

We look forward to continuing to contribute to projects that make such a positive impact like this project on the mobility and sustainability of the Netherlands!

Would you like to know more about this project or are you interested in working together to establish a successful CRO strategy in your organization as well? Please contact contact with us.