Peep Laja, oprichter van ConversionXL, haalde in twee recente artikelen een aantal uitspraken aan van Ton Wesseling en Bart Schutz over A/B-test statistieken.
Ton Wesseling over het belang van A/B-test statistieken:
“Within experimenting the trade off is always between exploration and exploitation. Do you want to know the real answer, or do you just want to make a profit?
If you are testing medicines that should stop someone from dying – and the A variant is not getting those medicines, when do you stop feeding Variant A with nothing when you see Variation B is working? During the test? After a couple of positive tests? How many people need to die before you decide to give everyone Variation B.
Of course running a business is less dramatic than the example, but still – you want to outgrow your competitors – so you want to learn, but not be fully 100% sure, because that will make you too slow – not adjusting rapid enough to user wishes.”
Ton Wesseling over statistical power:
“You want to test as long as possible – at least 1 purchase cycle – the more data, the higher the Statistical Power of your test! More traffic means you have a higher chance of recognizing your winner on the significance level your testing on!
Because…small changes can make a big impact, but big impacts don’t happen too often – most of the times, your variation is slightly better – so you need much data to be able to notice a significant winner.”
Peep Laja opende zijn artikel over sample vervuiling met een annecdote over Bart Schutz:
For example, at CTA Conference in Vancouver, Bart Schutz of Online Dialogue took the stage and asked 400+ marketers to raise their hands if they know what sample pollution is. Less than 20 people raised their hands. In Bart’s words, “If you don’t know about sample pollution, stop testing.”
Ton Wesseling over cookievervuiling:
“I only say you need 1,000 conversions per month at least because if you have less, you have to test for 6-7 weeks just to get enough traffic. In that time, people delete cookies. So, you already have sample pollution. Within 2 weeks, you can get a 10% dropout of people deleting cookies and that can really affect your sample quality as well.”
