Read the English version of the Sentient Ascend Case Study here.

Evolutionary Algorithms in CRO: the first case study in the Netherlands

The conversion optimization market is maturing. More and more CRO teams are looking for new ways to take their CRO program to the next level, and more and more new tools promise to make our work more efficient and profitable. Still, from a technology standpoint, it has been a while since a disruptive tool entered the market. Until Sentient Ascend made its appearance.

AI for greater efficiency

Sentient Ascend is an automatic conversion optimization system that uses an evolutionary algorithm. A tool for massive multivariate testing that cleverly tests out new combinations all the time, in order to find the ideal combination in the least amount of time. Instead of a single new variant, as in an A/B test, multiple variants are tested in parallel. In addition, the experiment is split into generations in which a limited number of possible combinations are tested each time. This in turn differs from a Multivariate Test in which several variants of multiple elements on a page are tested simultaneously.

Each new generation, the system learns which combinations with which element variations score best. The ‘winning element variations’ are combined in the next generation and tested again. In this way, the system searches for the best combination of element variations as quickly as possible, without testing every possible combination. If the element variants are properly designed, each new generation provides another uplift in the average efficiency of the active combinations compared to the original starting situation. Using parallel interactive evolution, Sentient offers a new way to conduct large-scale multivariate tests and to make optimal use of the time that normally elapses between completing, analyzing and starting a new experiment.

What makes a Sentient Ascend experiment different from an A/B test?

Conversion optimization is the “evidence-based” (re)design of customer journeys with the goal of getting people to perform certain tasks (for example, registering for a service or buying a product) as successfully as possible. To understand which hypotheses - set from behavioral research - have what effect, online experiments are used. Often you then try to find out through an A/B test whether the new adaptation on your site performs better than the current situation.

Setting up an A/B test takes time and for reliable test results you need enough visitors. How many experiments you can perform therefore depends on the number of visitors to your website and the time (resources) you have. In practice, therefore, we see that what you can test is often limited and must be prioritized well to ensure that you perform the right experiments.

For this reason, in our view, Sentient Ascend is a disruptive tool in the market - the tool, above all, wants more and more input. The more elements on a page or in a customer journey get a variant, the more possible combinations are created. More combinations theoretically provide a higher global maximum, and the system is designed to find this maximum as quickly as possible. The caveat, however, is that you still want to do good research to determine which variations you make on elements. Too much garbage as input causes the system to spend too much time filtering out the garbage, so you are not using the system optimally (so prioritizing hypotheses just remains a must).

The number of combinations that are tested within a generation does be limited by the number of visitors you have, but at least you are sure, unlike A/B testing, that you are fully exploiting your testing potential. You maximize the number of tests you can perform within your digital domain.

Proof of concept at Euroflorist

It didn't take us long to find a suitable CRO team who shared their enthusiasm for the new tool. Despite the fact that the team at the online flower store Euroflorist (at the time led by Guido Jansen) already has a solid CRO program, they are always open to innovation. Euroflorist is an online pioneer for good reason. In 1995 they sold the first online bouquet (and were one of the first 100 webshops worldwide) and now most of their revenue comes from online. From 19 sites (spread over 11 countries) they sell an average of 2 million bouquets per year. In addition, they have over 10 million visitors on their sites and achieve conversion rates of over 25%(!) on some sites.

In the spring of 2017, we - Euroflorist as test platform and Online Dialogue as creative and technical partner - rose to the challenge and welcomed Sentient Ascend with open arms for a proof of concept on Euroflorist's Swedish desktop website.

What did we want to learn from this Proof of concept?

1. In practice, is Sentient Ascend really a method for improving your conversion rate? Does it deliver on its promise?
2. When does it make sense to use this solution?
3. Is the winning combination a variant that we would have found on our own via A/B testing?

The design of the Sentient Ascend experiment.

The proof of concept started with an extensive hypothesis session. From Online Dialogue's ‘Evidence Based Growth’ principle, a determinants of behavior study was conducted at Euroflorist based on available behavioral data, previous (A/B) tests, surveys, heatmaps and scientific research on flower sales. During the hypothesis session, this knowledge was mirrored with Euroflorist's knowledge and hypotheses were prioritized (based on probability of success, substantiated by the available research data) and variations devised. Our bandwidth calculation showed that we could test 8 combinations per generation with the visitors of Euroflorist Sweden. We therefore chose a setup of 8 different elements, each with 1 variation per element. In practice it is possible to make even more variations per element, but for this Proof of Concept we chose a simple setup.

Based on the determinants of behavior research, the following 8 elements were ultimately addressed and provided with a variant - the hypotheses are not fully written out below, but the elaboration of the variant for each element is.

Header

This is the page where one makes the choice of which bouquet. The first moment when one also starts thinking consciously about the choice. The header is quite busy, making it less busy gives less distraction and more focus on what energy should be spent on.

USP Bar position

The visibility of the USP bar is not ideal; we created a variant where it was shown at the top instead of the bottom of the page

USP Bar Content

The motivational content bar features new information, information that we expect will provide more motivation to buy from Euroflorist.

Well/no progress bar

Visitors are not told what will happen in the next steps if they decide to buy, this can create ambiguity and that causes dropouts.

Price display

To ensure greater average order value, the additional price is shown per step higher in size rather than the total price

Social sentence under CTA

The Call to Action is viewed without question before proceeding. To remove fear, a short social proof sentence can be used to ensure that one is motivated to continue with the process.

Position of the product image/Call to Action block

In the Western world, we read/scan from left to right. By rather focusing on the price choice block and alternating it with the picture block, one sees the different variants first instead of the picture first.

Product information / warning

Directly below the Call To Action, visitors are warned that the book may look slightly different in practice than it does on the image. This warning is very close to the Call To Action and can cause abandonment because of the negative angle.

Survival of the fittest - the results and the winner tested

The whole experiment on the Swedish website ran for a total of 11 weeks, with 4 generations fully completed. In generation 3 we saw the first significant positive uplifts and the 4th generation had multiple combinations with a significant increase in conversion rate. In practice, we indeed saw an increase in the average conversion rate of all combinations in that generation compared to the original page for each generation.

The best performing combination after the 4 generations was a specific combination of 5 different elements with a relative uplift of 4.3% in conversions (transactions on the website, with no impact on average order value).

In the above best performing combination, the following things were adjusted:

1. Progress indicator at the top of the page
2. Price perception adjustment
3. Social sentence under the Call To Action
4. Product-warning information less close to the Call to Action
5. New USPs in the USP bar

We are almost certain that this is not yet the overall maximum, for which we would have needed several more generations. But given the run time and the fact that the system was beginning to show significant results, we decided to move on to the next step in the validation of the Sentient Ascend system: retest the winning combination with an A/B test to learn if the winning combination again beats the original.

This new testing in another A/B testing tool again led to a significant result (increase in orders, same order value) in a 2-week testing period (pre-calculated based on the expected uplift from the Sentient experiment). So the Sentient Ascend system was actually able to come up with winning combinations that were different from the possible winner we would have guessed: all elements active, possibly without rotating the product image and call to action block (which goes against existing usability of ecommerce websites in general). Our header modification and the combination of progress indicator and USP bar at the top had no positive impact.

In what ways would we use Sentient Ascend in a CRO program?

For us, A/B testing an sich is also exploration. We are a big proponent of first fully exploiting your test potential by scaling up in quantity and only then start working on more quality. Our learnings of customer behavior come for 50% from determinant studies (preliminary research) and for 50% from the analysis of executed A/B tests. The test analyses give us a lot of information about changes in behavior.

Because of its multivariate design, the Sentient experiment gives us less precise insights into behavioral change, but all the more insights into interaction effects and the right composition of elements. For us, the use of Sentient Ascend is ideal for optimizing a customer journey (cq. website, cq. (set) of pages) when you have learned through research and A/B testing what fundamentally works and what does not. If you know what winning hypotheses are then you can use these for an experiment in which you make an extra boost up by finding the ideal combination of elements.

Normally, after a series of winning experiments, we “re-align” the optimized flow - it seems logical to include Sentient Ascend in this step. In addition, Sentient is working on a segmentation update using AI: automatically recognizing matching behavioral segments and learning here which composition of elements works best. Once that can be done, “always on” really comes closer. The current setup is still a time-limited Sentient Ascend Experiment with a winner implemented. The future, of course, is that that won't be necessary: an experiment is running continuously, achieving more and more conversion and asking for new input when certain elements are proven not to make an impact.

Cost / Benefits - Time / Results

The Sentient Experiment takes much more time in terms of setup. You can reasonably compare each generation to 1 A/B test (in terms of lead time and time required), making the total time equal to a number of A/B tests. However, where in terms of A/B testing it is normal to find a significant effect 1 out of 3, or 4 tests, each Sentient experiment almost leads to a significant result, provided the right input is put into the system (which also applies to A/B testing of course). In our view, the right input is extra important with Sentient to make the business case for Sentient more positive: it is better at finding an overall maximum than a series of A/B tests, but then the system should not spend too much time on multiple elements that do not generate impact. Then it becomes less effective.

The cost of the tool is higher than many an A/B testing solution - the promise of growing from local maxima to global maxima should be able to make up for this. We also see Sentient Ascend as a good addition for our conversion optimization clients who are already busy learning about behavior and are ready to leverage these learnings more! Therefore, in addition to being Lighthouse CRO Agency for Google in the Netherlands, ABTasty premium partner and Optimizely partner, we are now also proud to be a premium partner of Sentient Ascend.

Want to learn more about what this way of experimenting can mean for your organization - contact Valentina Djoemai (lead of the Euroflorist Proof of Concept) at +3130 4100 177 or valentina@onlinedialogue.com.