Implications of big data in experimentation

In order to be as meticulous as possible, there are some guidelines that ensure we can do thorough research. The most important thing you start with as a researcher is to describe your expectation, or in other words you formulate a hypothesis. Through an experiment, you collect data to test a hypothesis. Now it is the case that we collect more data than is necessary to confirm or reject the hypothesis. This makes it incredibly tempting to use more than the required data for analysis. Especially in an experiment where we find no difference based on the hypothesis. Perhaps we found “something” after all?

The tendency after an experiment to look in the data for segments or conditions that make the hypothesis correct, or even to come up with a new hypothesis based on the data found, is common among companies experimenting online. But is it okay to make statements about data if you haven't hypothesized about it?

We too run into this question, which is why we thought about an experiment during Conversion Hotel 2018.

It is easier to make healthy, rational choices for yourself in the future than for yourself at the time you actually have to decide (Thaler, 1981). Conversion Hotel was the perfect opportunity for the psychologists at Online Dialogue to put this to the test. Time for an experiment!

The tangerine experiment

Attendees of Conversion Hotel were able to register for workshops through the event app. Registering through the app gave us the opportunity to ask prior to the workshop if participants wanted to eat a piece of fruit or a candy bar during the workshop. We did this under the guise of “we want to know how much to buy.” During the workshop, we told the participants that unfortunately something had gone wrong when passing on the orders. To make up for this difference, we had made sure there were enough tangerines and candy bars for everyone. So regardless of what you had previously passed on, you could now just choose what you were in the mood for. It was up to us to count whether more tangerines or candy bars were eaten.

Consistent with our expectation, a majority of participants (58 percent) chose the healthy choice, the tangerine, beforehand. By the end of the workshop, participants had eaten 59 tangerines and 45 bars. Which means that 57 percent of participants chose a tangerine (or the healthy choice) during the workshop. There is hardly any difference between them. We cannot confirm the hypothesis, that people make healthier choices for themselves in the future, than when people have to choose at the moment.

Too bad, end of experiment?

Not yet. We found out that there was a difference in when the snack was offered. In two of the four workshops the snacks were offered to the participants before the workshop, in the other two workshops they were offered after the workshop. If we compare those two groups we do see a big difference.

Of the group that was allowed to choose a snack at the start of the workshop, 76 percent chose a tangerine. After the workshop, this was 41 percent. This decrease in the number of tangerines eaten is significant (p<.0001). So: have we proven that people make healthier choices before a workshop than after?

The difference in the number of tangerines eaten immediately reminded us of the concept of “ego depletion.” In a famous study, Shiv & Fedorikhin (1999) found that participants who made a heavier cognitive effort were more likely to choose chocolate cake as a reward than a (healthy) salad. We can fill in the heavy cognitive effort in our experiment by the workshop. In other words, participants were quite cognitively exhausted after our workshops, which made them less able to persuade themselves to choose the healthy snack.

Nice, we have a successful experiment after all! Admittedly not confirmation of our hypothesis, but very nice to see that we inadvertently influenced the groups.

What goes wrong here?

Searching for your own rightness after the fact cannot be done. If you cannot give a (convincing) reason why your hypothesis is true under certain segments or conditions, you also have no reason to believe that the effect you found is not based on chance. Do you do have a good explanation? Then it can serve nicely as a hypothesis for your next experiment. Even though it is very tempting to say that it is a successful experiment because a difference was found between the groups.

Still, it is wise to be careful about post-experiment analyses. After all, if you do enough analysis, you will automatically significant effects find. And confirmation bias tempts us (often without realizing it ourselves) to see mainly evidence that confirms our expectations.

Looking at the results of our tangerine study, it seems quite plausible that the time when the snacks were offered had a significant influence on the decision. So nice start for a follow-up study!

1. Baba Shiv, Alexander Fedorikhin; Heart and Mind in Conflict: The Interplay of Affect and Cognition in Consumer Decision Making, Journal of Consumer Research, Volume 26, Issue 3, December 1, 1999, Pages 278-292, https://doi.org/10.1086/209563
2. Thaler, R., & Shefrin, H. M. (1981). Journal of Political Economy, 2, 392 - 406.