Insights from Conversion Boost 2024

Last month I attended the Conversion Boost conference in Copenhagen, an event focused on optimization and experimentation. For me, this was a special event for two reasons. First, I had not been to this event before. Second, I consider Copenhagen my second hometown (with Amsterdam coming in second) because I studied here and still like to return regularly to visit friends.

Below I share key insights and lessons learned from this event.

Humanity in optimization

During her presentation “Finding your humanity in optimization,” Abi Hough used Keller's Brand Equity Model to explain the essence of brand equity. This model is composed of four layers and emphasizes the importance of a deep understanding of your audience. This is essential in both brand equity and optimizing your digital products:

• Salience: Thorough insight and understanding of your audience is crucial. Go beyond the usual research and immerse yourself in their world. Abi illustrated this by making personal visits to garden stores to directly observe customer behavior.
• Meaning: Does your product meet customer needs? For example, is it accessible, simple and customer-friendly?
• Response: This concerns the perception and emotional responses your brand evokes.
• Relationships: Building long-term relationships, such as loyalty and community spirit, plays the main role here. An excellent example is Lego, which embraces product ideas from their community.

The financial realities of experimentation

Another insightful presentation was given by Bart Schutz, founder and owner of Online Dialogue. He spoke about the financial implications of experiments. He highlighted a common problem where reported business cases are often inaccurate and far too high. This is due to:

• False positives: When an A/B test is a winner, but in reality makes no difference.
• False negatives: Situations where tests are not winners, but there is a positive effect in reality.

Bart indicated that growth and impact can be increased by:

• Generating better ideas.
• Optimizing significance to improve the balance between false positives and false negatives.
• Increasing the power of experiments.

Want your boss to become a fan of experimentation despite seemingly disappointing results? Then report a statistically correct business case. 

We are also launching a new master class at Online Dialogue: The replication crisis of unreliable A/B test winners.

The potential of failed experiments

Daphne Tideman devoted her presentation to the importance of learning from failed experiments. She argued that failure provides an important opportunity for growth and improvement, and introduced a five-step plan for turning failure into teachable moments:

• Make sure testing is part of a broader business strategy.
• Conduct a pre-mortem with stakeholders.
• Align with your stakeholders to address their concerns in advance.
• Take time for a comprehensive post-mortem.
• Use the insights gained to inform future experiments.

Culture of experimentation at major brands

It's always nice to hear stories from organization where there is a culture of experimentation. At Conversion Boost, experts from Ikea, Spotify, and Sky shared their stories.

Arnoldo Cabrera of Ikea shared three lessons about starting and running an experimentation program:

• People are the best asset: Technology and tools are becoming cheaper and accessible, but access to talent remains limited.
• Experimentation is a game changer: all you have to do is start. Get a sponsor, data and people without opinions.
• Start small, think big: start with minimal technology and team structures, but always with a vision of scalability and innovation.

Ashit Kumar of Spotify spoke about optimizing Customer Lifetime Value (CLV):

Online experiments must take CLV into account to ensure they do not harm the OEC (Overall Evaluation Criteria). The OEC is a combination of data that can help make decisions important to the long-term success of the company. In other words, short-term gains should not be harmful in the long run. 

It is challenging for many organizations to accurately calculate CLV for different cohorts because CLV is long-term; therefore, reliably determining cause and effect is challenging.

One solution is to find proxy metrics that indicate long-term retention. For example, for most apps, this is user activity on the platform.

Finally, Natasha Senior of Sky spoke about setting up an experimentation program to reduce operational costs. Herewith her tips:

• Link your KPIs to business objectives so that your strategy is aligned with the rest of the organization from the beginning. So make sure you have a KPI that aligns with the business and that people understand.
• Adapt the experimentation process to fit the company and its operations.
• Don't keep experiments to yourself: educate and adapt to business needs.

Strategy & people-focused optimization

These real-life stories highlight the importance of a strategic approach and show how companies can be successful by combining both data-driven and human-centric approaches in their experimentation processes.

Conversion Boost was a valuable conference in a beautiful location with excellent speakers and therefore definitely worth repeating.