As a company, how do you make the decision to purchase an A/B testing tool? And what are the differences between them? Here is an initial outline to get you started.

The landscape of providers of testing tools may not have expanded in recent years, but it certainly has if you look at the possibilities these tools offer. The search for a suitable tool could therefore be called an experiment in itself.

For this search, you can take a lean approach. Just try out some tools and experience for yourself what suits you and your organization best. Or you get good and extensive advice so that you not only find out what the practical experiences of others are, but also what other options there are.

Researching A/B testing tools

Forrester recently released a new report in which they compared several tool providers. How these tools are positioned in the market can still differ. There are providers who still call their product or service an A/B testing tool, but by now most of the market has evolved considerably and many tools have become increasingly complete platforms for optimization.

They are platforms that now have extensive personalization capabilities in addition to the most well-known A/B testing capabilities or feature flags and rollouts offering. A welcome addition as far as I'm concerned! Especially for organizations that are already a bit further along in their test maturity, optimization has long gone beyond just the point of conversion. Partly for this reason, you have been seeing the term CXO emerge alongside CRO for some time now. Conversion optimization as a field of study is thus being pulled a bit wider again. With Conversion Experience Optimization (CXO), the emphasis is a bit more on optimizing the overall customer experience. The fondness (XO!) for conversion & optimization now seems to have been definitively passed on to other disciplines such as UX and marketing as well.

Different features

Optimizing the overall customer experience may sound like (n)something new. Providers of tools and platforms are always cleverly capitalizing on emerging trends and developments. They naturally want you to have a all-in-one solution is going to work and optimize for that entire customer journey. They offer you that more complete solution to work even faster and more efficiently ... they say. No separate tools but serving all digital touchpoints from one environment. Think of a kind of toolkit with all kinds of useful tools. Sounds ultimately pretty handy right? Taking even bigger steps. Or quick lots of small steps....

Especially faster because you are able, for example, to launch new product features in a controlled way for a part of your customers and to gradually increase the size of this group (this is also called ‘ramping up’). At the same time, new features can be switched on and off remotely. This is then done without the intervention of IT or developers and thus without deploying new code. This saves an enormous amount of time and this is exactly what these optimization platforms will continue to work on in the future: further automating the experimentation and optimization process. The easier it gets for us, the better. Bottom line, it's only about one thing: “Fail fast, learn faster!”.

Personalization is also something that most platforms today offer to a greater or lesser extent. It has now become easier to personalize an A/B test, but also to validate a personalization as an A/B test. In my opinion, this is a very good development because personalization is still often seen as the ‘holy grail’ is presented while validating hypotheses through a controlled online experiment still provides the most reliable evidence. The combination can only improve the process and ensure that you make the right choices with even more certainty and put them live. 

Experience Optimization Platforms (EOP).

Meanwhile, Forrester is also talking about Experience Optimization Platforms, called EOPs for short. They conclude in their report that of the ten providers they surveyed, support for more digital touchpoints as well as use of AI will ensure they continue to make the difference. This means, for example, that rule-based optimization will become less effective over time. Furthermore, offering the right innovative optimization techniques, support for omnichannel campaigns and profile data management ensures that these providers will not only make the difference, but will also be successful at it in the future. But yes, that does come with a hefty price tag for the customer, of course. According to Forrester, Adobe and Salesforce are the leaders in this EOP market, followed by companies such as Oracle, SiteSpect and relative newcomers such as A/B Tasty and Optimizely.

Comparing A/B testing tools 

I've made a small selection below of tools that I've heard positive things about or because we at Online Dialogue often encounter them with our clients. These are clients that operate at various levels. These can be organizations that are just starting with A/B testing or organizations that want to further professionalize their current processes and methods. These are also increasingly organizations that have grown significantly in recent years and are thinking about taking the next step. For example, by developing their own (server-side) testing platform. This is a pretty difficult choice for many organizations. That is why Online Dialogue is regularly called in as a sparring partner and/or advisor in this phase.

The latter option is still a bridge too far for many. In fact, most tools meet today's needs and requirements just fine. They still do what they need to do. Below is an overview of A/B Tasty, Convert, Google Optimize, Optimizely and VWO.

Choose the right A/B testing tool for your organization

As you must have noticed, there are many similarities between the tools mentioned. That doesn't immediately make a choice easier. Therefore, also look at the organizations behind the tools. What is their mission statement? What do they consider important? How do they see their product development and do they have a clear roadmap for the coming quarters? But also: how is customer support arranged and does it fit my personal needs?
My advice: choose the tool that best fits the goals you have and the organization's test maturity, and don't be too quickly seduced by all the extra features that are potentially out there. It is more important that you are improving, then by what means you are doing so. Something that looks nicer and sounds bigger is not always better. Size doesn't matter, remember? It's how you use it!