CRO is increasingly becoming a standard in organizations. Within the Netherlands, most medium and large organizations have one or more CRO specialists, or a multidisciplinary team.

Organizations are also beginning to take the next step: Placing CRO specialists within product teams. This stems from the success of CRO, but also from the understanding that experimentation is indispensable within product and innovation teams.

Books such as Continuous Discovery Habits and Testing Business Ideas contribute to this. The reach and message of Marty Cagan (the most well-known thought leader for tech product management) also play a role.

There is just one very big difference between standard CRO work and experimentation within product teams. The traditional focus on A/B testing does not always meet the needs of innovative projects.

It is, for example, inefficient to invest months of time and resources in developing a big new idea, only to discover that the A/B test does not produce the desired results. This can lead to frustration within teams and waste valuable resources.

In this article, I discuss why CRO specialists need to expand their methodologies to include various validation methods such as user testing and how we approach this at Online Dialogue. This broader approach allows CRO specialists to better collaborate with product and innovation teams, validate innovations step by step and thus broaden the scope of experimentation.

CRO specialists have the right philosophy. Now is the time to expand our standard approach and make a bigger impact.

Current limitations of A/B testing

For many years, A/B testing has allowed us to reliably validate ideas and hypotheses through controlled experiments. However, this methodology has limitations, especially when it comes to the needs of product and innovation teams and their processes.

1. Limited suitability for innovation.

A/B testing is ideal for optimizing existing elements, such as improving copy, a call-to-action, or changing a layout. But when it comes to large-scale innovations or introducing entirely new features, this method often falls short. It is not realistic to develop a completely new product feature and validate it only with an A/B test. We eventually want to A/B test, but before we put development to work for a long time, we want to know whether the innovation or feature has a chance of succeeding and how.

2. A/B testing can be too slow in some situations

At the start of the development of a new innovation or feature, you want to run validations at a rapid pace to test assumptions and gain insights. Because A/B testing relies heavily on quantitative data with a lead time of one or more weeks, it can slow down the discovery and validation process.

At Continuous Discovery Habits Teresa Torres explains in great detail: “The way most teams test ideas isn't feasible when working with a set of ideas. We can't build three ideas for the same target opportunity, and A/B test them to see which is the most effective. It would take too long. Instead, we need to learn how to quickly test our ideas through fast iterations.” ... “Rather than starting with a large-scale experiment (e.g., surveying hundreds of customers, launching a production-quality A/B test, worrying about representative samples), we want to start small.”

3. A/B testing is not always possible

When there is insufficient traffic and conversions, or when the platform does not yet exist, A/B testing is simply not possible. Still, you want to validate assumptions and ideas and will have to look for alternative methods.

Various validation methods

In the book Testing Business Ideas several validation methods are introduced that can be used to test hypotheses and validate ideas. These methods can be divided into two categories: discovery and validation experiments.

Discovery experiments

Discovery experiments are designed to explore new ideas and opportunities. They help teams better understand customer needs, test concepts and discover which direction is promising. Some examples of discovery experiments are customer interviews, search trend analysis, online ads, social media campaigns.

With discovery experiments, you find out if the direction of the innovation or product (feature) is correct, test assumptions and the direction of the innovation or product (feature).

Validation experiments

Validation experiments aim to confirm whether an idea actually works and is viable. These experiments help test assumptions and minimize risks before major investments are made. Some examples of validation experiments include: Clickable prototype tests, simple landing page, presale and, in some cases, A/B testing.

The benefits of various validation methods

Using a broader set of validation methods offers several advantages.

1. Quick and valuable feedback

By deploying discovery experiments such as short interviews, prototype usability testing and 5 second testing, teams can quickly gather feedback and implement iterations. This speeds up the innovation process and helps get the right ideas to market faster.

Through familiarity with different methods, you can choose the right method for the right and fastest insights, appropriate to your situation.

2. Reducing risks

Both discovery and validation experiments reduce the risk of putting unnecessary resources into an innovation that has no chance of success. They allow teams to test and refine assumptions to increase the likelihood of success before making large investments in product development.

3. Deep understanding of users

These methods offer many insights that help to better address customer needs and develop more effective products.

4. Flexibility and adaptability

Using various validation methods makes teams more flexible and better able to respond quickly to changes in the market or customer needs. This increases the likelihood of success in new product introductions and innovations.

5. Integrated approach

By combining discovery and validation experiments, teams create an integrated approach that includes both exploration and validation of ideas. This provides a more holistic and effective way of experimenting and innovating.

The process

The philosophy of experimentation and optimization remains the same whether you are working on a big new innovation or a small copy test (based on research, of course). What may vary, however, is the process you follow. This process depends on the nature of the project and the specific goals the team wants to achieve. For a large innovation, the process may be more extensive with more different research and iterations, while for a small product modification, the process may be simpler and shorter.

Many organizations already use processes such as product discovery (with Opportunity Solution Trees), the double and triple diamond methods, and dual-track agile. These processes are designed to help teams explore opportunities in a structured way, develop solutions and continuously validate assumptions.

At Online Dialogue, we use these processes as well, but we add our own 15 years of experience and expertise. Our approach is unique because of our experience in validation and experimentation. In addition, we apply the combination of psychology and data at every stage of the process. This allows us to determine not only what the outcomes of experiments are, but more importantly why these outcomes occur. These deeper insights allow us to make more informed decisions and achieve even more successful results.

Are you joining a product or innovation team as a CRO specialist? Then try to understand the current process there as well as you can. Then reinforce the process and working method with your knowledge and philosophy of experimentation. That way, together with the team, you will achieve the best results and reach the goals you are working on together.

Increase your impact

To make a greater impact and better collaborate with product and innovation teams, CRO specialists must be willing to expand their methods and processes. By adapting our methods and using discovery and validation experiments, we can better support product and innovation teams. With different validation methods, we realize insights faster about the opportunities of an innovation or new product, what it should look like, reduce risk and get valuable results for the team and the organization faster.

If you need help setting up experiment-driven practices in product teams or innovative processes, feel free to take contact with us.