On Thursday, Dec. 8, 2022, were Ton Wesseling and I were guests at Experimentation Elite in London. Experimentation Elite is the only conference in the UK entirely about CRO & Experimentation. It is a one-day event with 11 speakers and a panel discussion. Ton was one of the keynote speakers at the event. 

In this blog post, I discuss the five key insights I gained during Experimentation Elite 2022 in London.

1. Conduct meta-analyses to gain insights from your entire program (Stephen Pavlovich)

The learning from experiments is a topic we at Online Dialogue have been working on a lot lately. Stephen Pavlovic of Conversion.com recommends conducting meta-analyses to get insights from your entire CRO program.

Although A/B testing is a reliable form of research, an A/B test can be subject to statistical error. Also, experiments can result in a winner, but for a different reason than stated in your hypothesis. This is why organizations should conduct meta-analyses (most reliable form of research). In a meta-analysis, similar A/B tests on the same pages are taken together to get a good idea of what works and what doesn't.

Three steps are required to arrive at a meta-analysis:

• Experiment framework: standardize the categorization of experiments and results.
• Good documentation: Conversion.com and Online Dialogue use Airtable for this. Here is a free online course on Airtable. It works easily in large projects and it gives a lot of overview.
• Automate: Automate everything to make it faster and more efficient. Another benefit of Airtable.

2. Experimentation should be part of the engineering pipeline (Stewart Ehoff)

Data from Optimizely shows that there is a big gap between companies doing 1-20 experiments per month and those doing 100+. The big difference between these companies: At the 100+ experiments per month, experimentation is in the engineering pipeline. At the other companies, CRO is in a separate team.

How to get experimentation in engineering:

• Get the technology right (server-side rather than client-side) and give engineers full access in the process.
• Make experimentation as easy as possible by making resources available.
• Speak the language of engineers.
• Find advocates (those who are already enthusiastic about experimenting) and give them a voice.

When you get this done and experimentation comes into your engineering pipeline the quality and speed of experimentation goes up significantly.

3. App testing is important and possible, despite some technical challenges (Gabriela Szpalerska)

That experimentation in an app is important goes without saying. Yet many organizations don't start it yet because of the technical challenges.

More development effort is needed, while they already have a full backlog. In addition, an app release must be done, which users must download, in order to put an experiment live.

There are more and more engineering solutions for setting up experiments without release. Another solution corresponds to point two above: get experimentation into the engineering pipeline.

4. Words matter (Eden Bidani)

Through three cases, Eden Bidani showed how the proper use of text on different pages resulted in major improvements in key metrics. To arrive at these areas of improvement, she used five questions in each case:

1. Where does the traffic come from?
2. What state of the buying process appeals to the current copy?
3. How difficult/easy is the text to understand?
4. Does the user have enough information to convert?
5. What is the users next step?

In answering these five questions, you'll quickly find out what's wrong with your current copy and how to improve it.
Listening Tip: https://www.marketingspark.co/in-a-fast-moving-add-world-words-really-matter/

5. Useful tips for achieving more AB test winners (Ton Wesseling)

When an organization starts experimenting more, the number of winners goes down. The main reason: quality goes down, not enough risk is taken.

Ton says, “Validation is not a solution to avoid risk. It's an invitation to take more risk.”
If you want to get more winners from your experiments, you need to take three steps:

1. Understand the entire customer journey (including the business model, competitors, alternatives, and scientific research)
2. Know where to optimize (thorough data research)
3. Create promising changes with different types of experiments (transactional, informational and transformational), different psychological strategies, based on meta-analyses.
When you apply this well, you are going to find more winners, learn more and realize growth.

Experimentation Elite 2023

The next Experimentation Elite is scheduled for June 28, 2023. At this edition, the roles are reversed: Ton is a guest, and I am a speaker. This edition will also feature the Experimentation Elite Awards. Previous editions we fell in prizes And we will try hard to win again next year!

In short, it was a very successful event and we look forward to the next one!