Join our free event Dialogue Thursday on November 13. Sign up now!
The past few weeks for me have been all about giving presentations, hosting panels, conversations in the corridors, and getting information online at five leading events: Convers(at)ion, Experimentation Elite, Growth-Minded Superheroes, Experimentation Culture Awards, and the Women in Experimentation Summit. Five events, five venues (two of them online), dozens of conversations, and one clear conclusion: our field is making mature strides.
Of course, each company has its own context. But six clear themes came back. In this blog, "6 Trends Shaping Experimentation in 2025," I summarize them, share a few quotes from speakers that have stuck with me, and give a suggestion for experimentation for each trend.
What stood out: more and more companies bring experimentation into the home. No longer as a project for "the CRO team," but as an integral part of product teams. Analysts are increasingly collaborating with designers and developers and are increasingly on the same product team. This makes feedback loops shorter and insights can be picked up immediately.
Also at Online Dialogue we see this development: more and more often we fulfill a role as strategic partner. We not only help organizations with the execution of experiments, but actively think along with them about the right preconditions - from culture and structure to strategy. In this way, we ensure that data-driven work becomes permanently embedded in marketing and product teams throughout the organization.
This is reflected in our work for clients such as NS and DPG Media.
Things are also changing at the board level. Successful programs increasingly have a C-level sponsor that protects budget and positions learning as a strategic asset, not a side-project.
For those engaged in experimentation in an organization, it is therefore increasingly important to be well attuned to the various colleagues, stakeholders and interests in the organization. This was also what my own presentation was about at Convers(at)ion and Experimentation Elite.
"Tune in with your colleagues, just as you tune in with your digital users." - Ruben de Boer
Try this:
Map your stakeholder landscape. Who supports experimentation, who is against experimentation, and with whom do you have yet to build a relationship?
Several speakers spoke about the shortcomings of chasing quick profits. The conversion rate remains important, but more and more teams are now designing their experiments on much more sophisticated long-term goals, such as long-term customer value, predicted CLV, retention and intention.
The wrong measurement of success can also create unwanted behavior internally. Once you choose one number as a target, people may optimize that number, rather than the customer journey. In his presentation "You are what you measure," David Mannheim gave a great example. In India, many people were dying from cobras. To reduce this, the government gave a reward for each dead cobra, with the result that people started breeding cobras and killing them to make money.
"When a measure becomes a target, it ceases to be a good measure." - David Mannheim
Fortunately, at Online Dialogue, we receive more and more questions about proper success measurement and how it relates to customer behavior. As a result, we are increasingly concerned with goal trees and Opportunity Solution Trees and its method of operation.
Try this:
Create a simple Opportunity Solution tree: put the business goal for your team at the top, the opportunities (customer problems or opportunities) that affect that goal below, and the matching solutions. Review that tree quarterly.
AI was in almost every talk, and that was no surprise. LLMs are now used for hypothesis writing, qualitative analysis, and even for generating synthetic "test users."
But: the human role is more important than ever. Several speakers cautioned against uncritically relying on synthetic data. Good programs treat AI tools as colleagues with clear tasks, code reviews and a permission matrix.
For organizations that want to achieve success with AI, there must be good policies in place. In addition, AI must actually solve customer problems or add quality and speed to existing processes.
Try this:
Nils Stotz had a nice tip in his talk on Experimentation Elite; organize prompt parties. Give a particular assignment and learn from each other's prompts.
With all the developments regarding AI, it is important not to lose sight of the user. User research remains very important, and will only become more so.
Earlier in this article, I talked about the shift from experimentation to product teams. Here user studies are even more important to quickly test assumptions within innovation and product discovery workflows. That this is an ongoing trend does not surprise me, at Online Dialogue we are increasingly involved in product and product discovery pathways.
"Use data science and AI - but start with a real problem." - Juliana Jackson
Try this:
Speak with your customer more often, such as on a monthly or weekly basis. Find out real problems and opportunities that contribute to business goals.
Many talks provided examples where collaboration results in better outcomes.
For example, Lucia van den Brink showed that test ideas arising from teams have a much greater impact than test ideas concocted by an individual. If a team is made up of diverse individuals (position, background, gender, etc.) this works reinforcing.
Also during the Culture Awards were weekly ideation sessions discussed, in which PMs, designers, copywriters and analysts collectively pitched problems (not solutions!).
Psychological safety is very important here. Teams that openly shared their "most interesting failures" saw an increase in the number of ideas as well as a higher win rate.
Personally, I am very pleased with speakers who share these findings. With my clients, I like to organize brainstorming sessions and involve colleagues. Not only for better results, but also to make more people aware of experimentation and to give everyone a chance to express their ideas.
"We need other people and other perspectives to come up with something new sometimes." - Lucia van den Brink
Try this:
Organize a brainstorm with a diverse group of colleagues about a customer problem or opportunity you have discovered in your research.
Accessibility is no longer a checkmark, but a moral obligation and growth opportunity. Also ethics and sustainability are being mentioned more and more often. These facets come up more and more often, including in ROI discussions and goals.
There is a lot to come on these topics. The European Accessibility Act that just went into effect, the use of AI and greenwashing and how we treat our planet was cited by several speakers and is also starting to become a growing topic within organizations. For experimentation teams, it is essential to be aware of and respond to.
"1 in 4 people need accessibility support and depend on accessibility of digital domains." - Shirley van Haalem
Try this:
Download the Chrome extension Wave sometime, and analyze how accessible your website is.
Trends remain abstract until you do something with them yourself. Pick one theme that fits your context, design a small change (on the website, in your team, or process), and test it. Measure the outcome and what you learn. Then share that story at your next team meeting. Don't forget to include your colleagues there.
Our field is making mature strides. Increasingly, we have the attention of leaders. Now it's up to us to ask better questions, align with different organizational goals, consider stakeholder needs, engage colleagues and connect with our customers more often and better. So we take the next steps together and have many insights and learnings to actively share in your organization.
Hopefully this overview will help you with that. Have fun!
This blog was written by Ruben de Boer, Lead Experimentation Consultant at Online Dialogue. Ruben helps organizations embed experimentation into teams and strategy, and regularly appears on national and international stages to share knowledge and insights about data-driven work.
EN
