Online experimentation costs time and money, which is why we want to do good experiments. The problem is that we don't know beforehand whether the idea we want to test is the right idea. The advantage of working in an online world is that there are multiple sources (data research, user testing, etc.) that allow us to assess whether our ideas have potential. We at Online Dialogue have recently been doing a lot of pre-validation of ideas through notifications.

Pre-validate

Pre-validation is investigating in the easiest way possible whether the A/B test will have a positive effect. This way, with minimal effort, we can increase the chances that the real experiment will be a success. Pre-validation is widely used in the medical and pharmaceutical world. For example, patients are given a light dose of medication to see if the drug has the desired effect before the prescribed dose is administered.

The advantage of pre-validation is that it can save unnecessary costs because you prioritize your experiments in advance for potential. This way, you increase the ROI per A/B test.

Pre-validation as input to your prioritization

Pre-validation can be done in many different ways. Of note, we want the studies to score as high as possible qualitatively on the ‘hierarchy of evidence’ (see image below). The higher the study is in the ‘hierarchy’ the greater our confidence that the results found are actually true.

An example of a pre-validation is asking a passerby what he is missing on a page. But an A/B test with more than 1000 conversions is also a pre-validation. You can probably estimate that the question to the passer-by scores lower than the A/B test on the hierarchy of evidence. Should the passer-by and the A/B test point to the same thing, this will strengthen your pre-validation. In other words, more reasons for you (and your manager) to conduct this experiment!

Notifications as pre-validation

Currently, we use many notifications to pre-validate. This is a simple method where we overlay a piece of code over the site. All we do is slide a notification into view and make it disappear again. The advantage of this is that it takes little development capacity, it can be implemented quickly and we don't have to take into account the current design of the website. In addition, we are pretty sure that visitors see the notification (it moves) and in the piece of text we are able to test a (psychological) principle.

For now, we assume that if a pre-validation has a positive effect, it is more likely that we will also find a positive effect in our A/B testing.

By working in this way, we can maximize our testing capacity to test good ideas and eliminate less good ideas in advance. To ultimately increase the number of successful experiments.

Pre-validation in development

Of course, there is room for development within this method that I would like to explore further. For example, if your pre-validation is negative, does your research stop? And to what extent can pre-validation predict the outcome of your A/B test, since a moving notification might give a completely different experience than a static design adaptation? In short, plenty to work with.

Do you use pre-validation methods to test your test ideas in advance? I find it very interesting to hear your experiences.