DPG media is the largest media company in the Netherlands. Through radio, television, newspapers, magazines, sites, apps and podcasts, they bring out news and entertainment. A dynamic field of work, but what about the culture of experimentation at one of the largest publishers in the BeNeLux? We ask Joshua Kreuger. Joshua is responsible for the experimentation program of the subscription department of DPG Media. With a team of 8 specialists, he focuses on setting up and improving the validation-driven process. This year his team has been nominated for the Experimentation Culture Awards and together with Online Dialogue as Best Agency team at Experimentation Elite, and for good reason. In this interview, we discuss the ideal recipe for a successful experimentation program. 

CRO is addictive

According to Joshua, CRO is the most fun field there is. 'CRO combines data, psychology and sales. And you can use your creativity in doing research or coming up with special solutions. The energy boosts from the 'aha' moments are hugely addictive. The eureka! feeling when a hypothesis is confirmed - or turns out to be just the opposite of what you expected - is contagious. It makes colleagues from other disciplines want to work with data.'

The importance of experimentation for DPG Media

DPG Media is a growing organization and is omnipresent in the lives of most residents of the Netherlands and Belgium. Be it through news titles or products such as Independer, a comparison site for financial products, health institutions and energy suppliers. 'Especially in the publishing branch, digitalization and the creation and capture of digital value is a gift, but also a big challenge. It is precisely through validation and experimentation that we can determine the direction in which to innovate and optimize. And the fact that colleagues are receptive to experimentation is not surprising. Wanting to know how something is really done is the DNA of journalism. And you then see that reflected in the other ranks of the company.'

The ingredients for a successful experimentation program

For the past few years, Online Dialogue has been working with DPG Media to establish a center of excellence for a successful experimentation program. Our maturity scan shows that the organization is in the structure up phase in 2021. This is the phase where, among other things, there is a team running data-driven experiments and a defined process has been established. Meanwhile, it is growing rapidly toward the Maximum Growth Efficiency phase and we are working hard to achieve that. To get to this phase, the team is growing in different areas. In the image below, you can see what factors play into the growth. Online Dialogue supports DPG Media's CRO specialists through coaching, training and developing smart tools and playbooks.

Joshua says, "A multidisciplinary team is obviously essential for a strong program. CRO is a super multi-faceted profession. For a successful experimentation program, it is necessary to have the right expertise at the table. Web analysis, psychology, UX design and also copy are, in my opinion, indispensable if you want to take a bigger approach. Because we lacked that knowledge ourselves, we sought collaboration with agencies. We have been working with Online Dialogue for a couple of years now and that has not been a bad thing. They are very strong in data (science) and psychology and we needed a clear vision and method to get the organization on board.

Experience shows that a good CRO program starts with someone who is passionate about CRO or validation-driven work. An ambassador who dives into the data, knows how to take people into the customer's problems and dares to question the current state of affairs.In terms of resources, still, you often see programs develop focus around a few tools or data sources and are invested in such a way. I think the choice depends in part on the development mode of your program or organization. If you want to innovate faster, then you are more likely to do prototyping and UX studies. If you're more on real optimizations then you'll probably stick closer to web analytics and other quantitative tools.'

From experimentation to validation culture

While working with Online Dialogue, the process and vision for a successful experimentation program was formulated and promulgated. So how do you move to the next step and create a validation culture? 'DPG has already more than embraced validation. I often see colleagues outside our team coming up with or setting up experiment proposals, which tells me that the value is recognized. Validation Culture I find it a difficult concept - opinions are also quite divided about what it entails - but I see, for example, that we rarely put things live without validation and that there is also social control of this by colleagues. That's quite a lot if you ask me!'

'We wanted to make colleagues and management part of the validation process in a way that matched their goals. In doing so, we focused mainly on people who were receptive to experimentation and did not spend too much time on colleagues who were less enthusiastic. In this I was fortunate to work in a fairly flat organization, a director who found experimentation interesting and that in Niels Rewinkel I had a first teammate with whom I was very complementary in character and skills.' [Niels Rewinkel was Experimentation Program Manager at DPG Media, serving as DPG Business Manager SHOPS and Marketing Positions since 2022, ed.]

'In addition, my goal has always been to capture our experiments and programs in business value, especially through business cases. You have a budget to account for, then value should be extracted from that - or at least told how much was saved by poor choices not to make. Otherwise, next time, those euros will go to a campaign whose ROI is immediately apparent.

Testing for the sake of testing

The growth to validation culture did not happen without a struggle. A culture change is sometimes very elusive. The biggest challenge has been to get colleagues on board while keeping the focus on why you are doing it in the first place: to create a sustainable validation-driven working method. 'It's hard to explain why you want to validate so many things. "This is testing for the sake of testing" is a reproach I have sometimes received. You can only blame yourself for that - because as far as I'm concerned you haven't told a good story about why validation is almost always the better option. The support from Online Dialogue has been valuable in this, in the form of workshops and daily support with which the story of validation suddenly got different faces. You also have to present yourself with a mirror. In the beginning I got stuck in my own tunnel vision and the idea that everyone thinks what we as a CRO specialist think is important. I now understand that for many colleagues there is more that they want to get out of their work than just uplift or a nice business case. I have learned to empathize better with others to determine what they can get out of experimentation, and I believe we are pretty well on our way there.

How do you ensure that experimentation becomes part of the culture of the organization?

For a successful experimental program is culture very important. 'I think it's all about making validation relevant to others and understanding what are really the problems the customer is facing. I think that's very situational, but with us we've had a few tipping points of which opportunity-driven experimentation did prove to be a very effective one for both acceptance and outcome. I shared this recently at Emerce Conversion & Analytics 2023. Especially about the how scope on which you experiment and involve the department in experimentation can make a big difference in whether colleagues catch on or drop out. Furthermore, proactivity is also important. Provide (unsolicited) substantiated input on an important topic in the department and you suddenly become relevant to another group of people who may not have had much to do with validation or experimentation. With us, legal now even sometimes comes with questions about experiment results. That's cool right?

The future of CRO

CRO is more than a marketing discipline. To emphasize that CRO is a method and a mindset, Joshua prefers to speak of Experimentation.

'I see that validation-driven work is increasingly accepted as the way of working in organizations. At the same time, I also see that the drive to deliver output is often at odds with validating that output. Statistics, experiment design, experiment criteria and so on are otherwise not obvious things. Therefore, I think we will continue to need ambassadors in the coming years to monitor the validation-driven way of working.With digital value even higher on the agenda, I expect that this is the time to really assert the added value of experimentation. We are now entering paths that perhaps no one in the publishing world has walked before. To then make a feedback loop of insight and results can then be hugely impactful to our direction and success. The specialists in the various branches of DPG are also increasingly seeking each other out. Exchange on process, tooling and results - all green flags that show me it has a bright future within the company. We continue to build toward that organization-wide validation culture.

In that process, Joshua's role has changed. In his role as manager of experimentation, he has a more facilitative role. In addition, he is the lead of the Centre of Excellence. How do you like that? 'I'm more on the process and tooling with the goal of having the specialists spend as much time as possible on innovation and optimization rather than the peripheral issues. I also get involved a lot in new projects around personalization and new product forms. The bump of knowledge we have built up and documented in Airtable comes in handy with that. The past five years have flown by but it feels like we've just warmed up!

We look forward to the awards show of Experimentation Elite on June 28 in London and hope to take home the coveted Best Agency team award. We thank DPG Media for the years of cooperation and look forward to the future of experimentation within this organization. We are proud of everything we have already achieved together. In case you got excited by Joshua's story, his team is hard at work looking for new colleagues. And if you want to know more about our collaboration and are curious about what we can do for you, feel free to take contact up. Wondering what trainings we've given at DPG? On our Academy you will find a nice range of relevant training and education courses that we also offer in-house.