At conferences on CRO, I often hear enthusiastic stories about setting up and structuring teams, getting the organization on board, increasing the reliability and speed of the optimization program and all the successful and less successful A/B testing.

Hardly do I hear organizations share how they store their insights and what they learn from the knowledge they build about their visitors' behavior through CRO. Whether it's not happening at all I don't know, but at least it's not being shared (enough).

And that's a shame, because that's exactly where much of the value of optimization can be found.

What is CRO?

The name conversion rate optimization actually does a disservice to what the process actually entails and especially, to what it delivers to you as an organization. Validating your assumptions through online experiments leads to more than just an increase in your conversion rate. This is just your short-term return.

In the long run, it provides you with knowledge about your customers' behavior. Indispensable information that allows you to anticipate possible changes in this behavior.

From CRO to behavioral model

But how do you find out why a particular variant does or does not perform better? How do you build knowledge about your customers' behavior so you can use it organization-wide later?

Based on my experience as a behavioral scientist at Online Dialogue, I can identify five factors that affect the quality and thus the value of your insights:

1. Structure

An open door, but very important when you want to build insights. By having a clear structure within your experimentation program, you avoid chaos and maintain the validity and reliability of your results. You are aware of the choices that are made, how they are made and by whom they are made.

This ensures that if something happens - for example, if there is a bug or a campaign turns on that affects the results of your A/B test - you can anticipate it or make the choice to reschedule the experiment afterwards.

2. Data quality

Without reliable measurements, you know nothing. You can't base your assumptions on anything, and besides, the results your experiments produce are not useful. Always make sure your data is correct, check it yourself or have someone check it.

Are you measuring everything you want to measure? Are all measurements shot in at the right time? And are your data structured in the right way? Read more about how to get your data quality in order in the article from Reinier Koolmees.

3. Make sure you test what you want to test

The optimization process is one big collaboration between different disciplines. One of those crucial points in the collaboration occurs when the design needs to be aligned with the hypothesis.

A hypothesis is basically nothing more than a statement containing the expectation you have about behavior. The design is the elaboration of this hypothesis in the form of a (small) adjustment that will or will not influence the visitor's behavior on your site.

In the mirror moments that take place during the translation of the hypothesis into a design, a number of elements are important:

How big do you make the adjustment?
Is your adaptation being seen by the right group of visitors and is the adaptation big enough to cause a change in behavior?

Do all design modifications align with the hypothesis?
Are there perhaps elements being modified that could have a different effect on your visitor's behavior and thus contaminate your results?

Sometimes we feel the need to adjust an extra element because it looks nicer or maybe because you found a mistake or bug somewhere. All these adjustments may or may not have an effect on your result.

Make sure your design is clean and really only one factor differs from the control variant. Only then can you say with reasonable certainty what caused the change in your visitors' behavior.

4. Look for alternative explanations

Have you found an effect? Then start looking for alternative explanations. Don't keep looking for confirmation but be the devil's advocate. Are there potentially other explanations for the effect you found? If so, test them!

Just like in science, you keep trying to disprove your own findings; your hypothesis is true as long as you haven't found evidence to disprove your assumption. This way, you gather much faster, much more and much more reliable knowledge about the behavior of your visitors (than if you are constantly confirming your own assumptions).

5. Re-test your findings

This point is somewhat in line with the previous point, but certainly no less important. Behavior is not only complex and subject to an incredible number of different (internal and external) factors, it is also constantly changing.

For example, an experiment you conduct in February may have a completely different effect when you conduct it in the middle of summer. The target group with whom you conducted the experiment may develop different preferences over time. Not to mention the small contextual differences that can be different for each person every second.

Therefore, make sure you keep retesting. First, to see if other factors have influenced your visitors' behavior, and second, to find out if and how your visitors' behavior changes over time.

From behavioral model to innovation opportunity

Taken together, the above points ensure that you gather new knowledge about your customer experiment by experiment. How does your customer make a choice, what causes affect this choice and what role can you as an organization play within this choice process?

In time, you will begin to recognize patterns in your client's behavior and you will be able to create a behavioral model that allows you not only to respond appropriately to your client's wants and needs but also to anticipate changes in these needs and wants.

They say that every product or service, no matter how popular, will at some point be overtaken by new developments. You can't prevent this, but the collection of insights that your experimentation program can provide at least ensures that you are one of the first in the market to be able to see and respond to these shifts.