A/B test scan

Online Dialogue's A/B Testing Scan provides an honest, clear picture of the accuracy and added value of your A/B testing program and, more importantly, it immediately doubles the effects of your A/B testing by recommending which recently discovered new experimental paradigms, metrics and test designs to apply

Optimizing ROI and impact

Experimentation is about discovering opportunities and minimizing risks. A/B testing is one of the most reliable and cost-effective ways to accomplish this. But even with A/B testing, risks remain and opportunities remain undiscovered. Our A/B Testing Scan clarifies the true effects of your current A/B testing. More importantly, we reveal how you can improve your testing practices to further reduce risk, discover more opportunities, and double the impact on growth in revenue and knowledge!

Redesign your A/B testing

Our A/B Testing Scan offers a unique opportunity to learn and apply the latest insights and developments in A/B Testing and double the added value!

1. Understand the real impact: Gain an honest picture of how accurate your test results are and how much your A/B testing actually contributes to organizational goals, such as revenue growth, customer satisfaction and customer insight.
2. Seize more opportunities: Get insight into the optimization opportunities you are currently missing.
3. Update your A/B testing paradigms: We identify and prioritize opportunities to increase the impact and ROI of your A/B testing, based on all the recent A/B testing discoveries and innovations!

The rebirth of A/B testing

Our A/B test scan begins by calculating the truth about how many of your A/B test results are correct or just incorrect and exaggerated. The results are usually both eye-opening and uncomfortable....
=> But have no fear, there is tremendous hope!

Ten years ago, the empirical sciences discovered the same uncomfortable truth about their scientific studies and called it ‘the replication crisis. Since then, scientists have innovated and optimized experimental designs and statistics (during - what is now called - the ’Renaissance‘ of the empirical sciences). The A/B Testing Scan from Online Dialogue builds on these new discoveries and innovations to:

• Apply new and improved test setups and designs (e.g., explore-confirm)
• Personalize your significance levels (rather than using a non-optimal standard α)
• Use previous data to strengthen the power of your tests
• Increase the speed of your testing (e.g., by increasing democratization)
• Optimize your budget allocation (focused on ROI enhancement)
• Improve and automate your next-best-test prioritization (e.g., using AI)
• And other revamped A/B testing paradigms (to overcome the current replication crisis...)

Temporary starting offer: only €5,500!

If you have our scan done in 2024, we are offering the A/B Testing Scan at a discounted rate. This is your chance to be one of the first to double the effect of your A/B testing at this special price. Contact us today!

“Note: The A/B Testing Scan pays for itself immediately: we usually find some ‘losing’ tests, which, according to the new A/B Testing standards, turn out to be winners!”

Contact us today

Don't wait any longer to take your A/B testing program to new heights. Contact us today to schedule the A/B Testing Scan and take advantage of this unique offer. Your success in optimization starts here.

Get in touch

The 5 steps of the A/B test scan

1. Kick-off meeting

We get to know each other and discuss the why, goals and procedure of the scan. We plan the meetings and explain the steps and required information.

2. Impact analysis

Of all your A/B tests, we analyze the following aspects:

a. Quantity quality; How many tests are performed and how accurate are they?

1) True success rate: what percentage of the optimizations tested actually produce an improvement, and what percentage actually have no effect? To do this, we conduct a statistical meta-analysis of all A/B tests (note that the true success rate is not the winner's rate).
2) Sensitivity: How many of these real improvements led to a winning A/B test, and how many led to a losing A/B test (false losers)?
3) Precision: How many of the test winners actually lead to a rise (and how many are false winners)?
4) Speed; How many tests were run? What were the success statistics? To what extent are teams autonomous in this area? What are the constraints that prevent more experiments? How many of the winners were implemented, etc.?.

The combination of quality and quantity gives a clear picture of the overall accuracy of your current A/B testing practices. How many real opportunities are discovered and how many real risks are minimized (upon request, we can also analyze by team/site/country).

b. Costs & Revenues.; We analyze the returns and costs of different types of test results. What are the actual returns of the ‘real winners’? How much risk and cost do the ‘real losers’ avoid? And what are the costs of incorrect winners and losers? We also apply the necessary statistical corrections to these analyses, and where we need to make assumptions, we discuss and agree on them.

Additionally, we can determine the cost of A/B testing efforts. How much time and money is involved? And what do those hours and costs entail?

c. ROI; We now have insight into a) how much and how accurate your A/B tests are, and b) how much they yield and possibly what they cost. This gives us a realistic picture of the impact and ROI of your A/B testing on your organizational goals (your real ‘business case’) and the factors that determine the impact.

3. Workshop

Optionally, we present the results of the impact analysis in an inspiring and interactive workshop. This session is designed to inform those directly involved in the implementation of A/B testing.

4. Potential Analysis

We then calculate the potential for improving the impact and ROI of your A/B testing, taking into account your current budget/hours and a relatively short 12-month duration (our comprehensive ‘Experiment Culture Scan’ is more suitable for long-term -term advice).

How much improvement is easily possible in each driving factor? More importantly, what would be the additional impact on revenue, profit, CX, customer knowledge, etc.? By calculating the impact of possible adjustments in this way, we create a reasoned and prioritized list of improvements that can quickly improve - probably double - the impact of your A/B testing efforts.

5. Consulting & Presentation

We present a summary of the analysis, results and insights within your organization. We clarify the current impact and ROI of A/B testing on your organizational goals. We also outline the potential for impact and highlight opportunities for immediate improvement. This presentation is aimed at managers and budget/product owners, but is also suitable for the experts on the team(s).

Want to know more about the A/B test scan?

“Let us double your optimization success!”

Send Ronald a message