I hope that every experiment I analyze has a positive outcome. But the harsh reality is that we at Online Dialogue have an average winner percentage of 30%. This means that the vast majority (70%) of experiments do not have the outcome we expected.

Yet every time, I hope we have the right idea and the hypothesis, and therefore our expectation, is confirmed. That bias can be a problem when it comes to analyzing your experiments. And that problem is also known as the confirmation bias.

Confirmation bias can easily creep into your analysis. You start an analysis and have a certain expectation that you are all too eager to prove. You start looking for data that support this expectation. Possibly because of that attitude, you use the wrong method or you happen to miss just the data that proves otherwise.

Confirmation Bias in Science

Confirmation bias causes you to not be open to information that rejects your expectations. Any time you are analyzing results, you are at risk of confirmation bias. It is therefore a well-known problem within scientific research.

So did Nature an experiment in which 29 scientific teams received the same information about soccer matches. Each team was asked to answer the following question, “Are dark-skinned players more likely to receive red cards than light-skinned people?”.

Some scientists found that there was no significant difference between light-skinned and dark-skinned players, while others found a strong trend of giving more red cards to dark-skinned players. In addition to confirming that the Nature study found that dark-skinned 30% players were more likely to receive a red card than light-skinned players, the final conclusion of the experiment was that confirmation bias was evident among the participants.

Confirmation bias in CRO

The same problem can arise when you analyze an experiment. You have a certain expectation of the result and go into the data looking for confirmation of your expectations. Because of that approach or focus, you may overlook another result, unnecessarily affecting the outcome of the experiment.

As an analyst, you are the judge of whether your hypothesis is confirmed or rejected. When confirmed, the winning variant will be put online, with all the effects that come with it. It is therefore incredibly important that your analysis not be influenced by your own bias.

How do you avoid wrong analyses?

At the Israeli intelligence agency, they apply the rule of the tenth man. The rule is as follows: suppose there is a group of ten people, nine of whom agree with a particular action, the tenth person's job is to challenge and question the current belief. The goal is to look at things creatively, independently and from a new perspective, thus reconsidering the status quo of the other nine people.

In Nature's study, the teams finally came together to discuss their methods and approaches and came to a unified result together.

There are a number of ways you can reduce the likelihood of confirmation bias in an analysis:

1. Create a complete picture and do not draw quick conclusions.
2. Seek a different perspective and question your current beliefs.
3. Always discuss your approach and analysis with another person.

I don't think you can ever completely protect yourself from confirmation bias, ultimately you will always, consciously or unconsciously, have some idea or expectations of the outcome. But when you are aware of the risk of confirmation bias you can protect yourself from it as much as possible.

At least the above approach has helped me and my colleagues avoid mistakes and I hope you will too. Sometimes you get too deep into the data and your own story that you miss other insights.

Happy analyzing