Despite A/B testing being a standard method in market research these days, few open source platforms offer this functionality. But last year, software company Intuit to announce their real-time, enterprise-grade A/B Testing platform called Wasabi going open source. Intuit Inc. develops and sells financial, accounting and tax software for small business accountants and individuals. Wasabi has been in business since the introduction of PlanOut by Facebook in 2015 the first open source platform for A/B testing. We had to test that!

What is Wasabi?

Wasabi is a real-time, 100% API driven, A/B Testing platform. The platform allows companies and individuals to run experiments on the web, mobile, and desktop for back-end and front-end, products and marketing. Intuit describes the platform as fast, easy to use (Wasabi can be set up locally with docker in 5 minutes and the entire implementation takes less than a day, they say), inexpensive, and scalable. Also, Wasabi protects user data while maintaining your competitive advantage, eliminating the need to hand over your sensitive data to 3rd party services such as Google Analytics or Adobe Analytics, for example. A big advantage is the flexible UI for managing your experiments.

Let's test

After setting up a local server and entering some test data, we set up some A/B testing to see if Intuit's promising description proved to be true. At first glance, Wasabi seems like a solid framework, providing an excellent foundation especially for server-side testing. A big advantage is that competitors or consumers cannot see that you are using the tool, as is often the case with other tools (Optimizely or Visual Website Optimizer, among others). However, you do have to set up all the tests yourself with code; a WYSIWYG editor is not available. We also missed a preview mode when developing the tests.

Visitor data is stored in-house. Because experiment data for each visitor must be retrieved from the Wasabi server, an on-premises installation seems the most logical solution. Wasabi claims the tool is fast (30ms response) and can handle ‘enterprise-grade’ much. But if you run the Wasabi server in the cloud, you run the risk of network latency effects that negatively affect the speed of your Web site or application.

Also, despite its promise for ease of use, the tool has a very high IT content: before you can get started, you have to set up many things properly before installation and updates and write code to get everything working (the platform is completely API driven). For our test we got the tool workable locally on macOS, but getting a production instance live is very complicated in our opinion. Because the tool is still in its infancy, documentation is also fairly sparse and information online is scarce. In addition, you are responsible for implementing and testing any updates yourself.

Our conclusion

At first glance, Wasabi seems primarily suited for companies with a large IT department that want to set up server-side A/B testing. There is a lot involved in using Wasabi properly. It seems especially suitable as a framework for companies that want to develop their own in-house A/B testing solution, where basic features such as a management interface and various A/B test options are standard. If we compare it with GTMTesting.com (Testing via Google Tag Manager), Wasabi is a lot more complicated and GTM works independently of the client's code and server. In our opinion, it is easier to use existing client-side solutions (such as GTM, Optimizely and Visual Website Optimizer) if you want to start testing quickly.

The tests were performed by Data Driven Designer Roy Schieving and Analytics and Optimization Expert Hans Nauta.

This article was published on Jan. 12 at Webanalists.com