Ever heard of The Paradox of Confirmation? Perhaps better known by the name of its most famous example, Raven's Paradox, by German philosopher of science Carl Gustav Hempel in 1965. The problem is as follows: when doing research, you want to collect as many observations (data) as possible to be sure of your findings. Testing a hypothesis cannot be confirmed or rejected on the basis of one observation.

The problem with doing research is that you can almost never access all the information available. Therefore, we assume that our findings are true until proven otherwise.

The fallacy

The Paradox of Confirmation assumes induction, a mode of reasoning in which, based on a set of observations, the finding is assumed to be the rule. The Paradox of Confirmation addresses the fact that every hypothesis is equal to the alternative hypothesis. As long as the hypothesis and the alternative hypothesis are proven we speak of truth. The example of Raven's Paradox states the hypothesis, ‘All ravens are black. The alternative hypothesis that goes with this is: ’All non-ravens are non-black‘. When collecting data, you want to make as many observations of ravens and non-ravens as possible.

Confirmation Bias

In principle, both hypotheses are true as long as all the ravens I have seen are black and all the non-ravens I have seen are non-black. The problem arises the moment someone sees a white raven. Do we then change the hypothesis or do we change the observation?

Exactly at this point is the problem of The Confirmation Bias: the tendency to look for information that matches our expectations and beliefs. Unconsciously, we ignore new information or twist the information so that it matches our point of view. From an evolutionary perspective, we do this because we are social creatures. After all, we are more likely to survive if we adhere to the same ideas as our social environment (and not risk social exclusion).

We can argue for ourselves that the hypothesis is still correct, by putting away the white raven as a white dove (quite apart from the fact that I have plenty of black things that are also a non-raven).

Anyway, you understand the problem. We humans are poor at viewing information completely objectively, because because of our knowledge and experience we are always biased.

For my work, it is very important to be aware of this. When we are in the midst of learning about visitor behavior at a client, we need to guard ourselves from The Paradox of Confirmation and The Confirmation Bias. Therefore, always be critical, challenge yourself and also test alternative hypotheses as much as possible.

This would be a nice ending to my blog. But: validation in every organization. The Confirmation Bias inspired me to conduct an experiment among my colleagues.

How I manipulated my colleagues: the experiment

One takes 20 colleagues, five A/B tests, two conditions and a survey (and a pinch of persistence to get all colleagues to participate).

Part 1: The Prediction Bias

The assignment in this survey was quite simple: the question to my colleagues was to predict which variant would win in A/B testing. I used the survey to show five A/B tests that we actually ran at one of our clients. The question in each experiment was, ‘Which variant wins on conversion? There were three answer options each time: A, B, neither.

The difference between the two conditions was in the way the introduction was written. In the control condition it was only told that five questions would follow, whereas in the experimental condition a brief background was first outlined. I then determined which answers most closely matched the information the participants would receive through this background (Confirmation Bias). So the big question is: Were my peers influenced by the attached background information?

Part 2: The Confirmation Bias

The second part of this research came to life when I realized that the study was not yet complete. Part 1 of the experiment only tests whether I can influence my colleagues' prediction through background information.

With The Confirmation Bias, however, one is expected to either adjust the hypothesis (externally seek an explanation) or try to put the observation in a different pigeonhole (that wasn't a white raven, that was a white dove). Thus, according to the theory, my colleagues - upon seeing the real outcomes - would begin to seek explanations that align with the background information (or, in other words, their own expectations or beliefs).

Fortunately, by now my colleagues were incredibly curious about the results, so the perfect opportunity for me to continue my research. I divided the group into a control group and an experimental group, ran through the A/B tests from the survey again, and for each test asked why they thought Variant X had won or why there was no effect. With the excuse “let's see if together we can make the quality of our A/B tests even better,” the discussion was conducted and the answers were written on post-its for each person and test. This way I could secretly observe if The Confirmation Bias was reflected in the behavior of my colleagues.

The results

The data analysis of part 1 was quite simple. I simply counted how many answers per condition were equal to the predicted answer based on The Confirmation Bias. And what turned out, the control group divided nicely randomly among the answers. In the end, 38 percent (for an expectation of 33 percent) of the control group chose the predicted answer. This percentage was, expectedly, higher in the experimental group. In this group, as many as 49 percent went for the answers that aligned with the background information.

Interestingly, the post-its from the second part of the experiment hardly differed between the two groups. The same kinds of words were used to explain (read: rationalize) why the A/B test would have a particular outcome. The difference, however, is in the discussion. The control group asked a mountain of questions about the behavior and wanted to find out what else was going on first. The experimental group, on the other hand, went somewhere else entirely to look for answers: “isn't subtle offering culturally specific?” (the background information indicated that subtle help was conversion-enhancing) or “the change in design is too small.”.

The experimental group experiences that real outcomes do not match expectations. The response is to look for an external explanation or to question its own design and experiment. In other words, it seems that the experimental group does not want to believe that the outcomes do not fit the current findings. The control group, on the other hand, has less guidance due to the absence of the background information and seeks the explanation within the available data of the experiment. Interesting!

Conclusion

I am very excited about my experiment: it succeeded in demonstrating on a small scale how impressionable we are. I should add that the study group is obviously far too small to make reliable statements, but the differences in behavior among my colleagues give me enough reason to conduct further research on The Paradox of Confirmation.

From my experiment, I learned three things:

1. First of all, my colleagues were on to me... After conducting my experiment, a few colleagues said, “I got nicely influenced by you.” Not good for my experiment, but my compliments to my colleagues.
2. Second, collecting respondents is really a lot of work. (In my job, I deal with online research and fortunately have access to a lot of data and reliable experiments.)
3. The third and most important lesson is that The Confirmation Bias is called The Confirmation Bias for a reason. We are very comfortable receiving confirming information. A little too pleasant even, which is why even in our work we tend to see only the results we want to see.

The only method to counter The Confirmation Bias is to occasionally just have someone watch you who doesn't know anything about the research yet. Then you'll find out soon enough if you too have become a victim of your own Confirmation Bias.

This article was published on Aug. 1 at Marketingfacts