You hear it more and more often, organizations that A/B test server-side instead of client-side. What does this mean? Why should you test server-side? What are the challenges? And what different solutions and tools are there? In this article, we'll try to answer these questions for you as best we can.

Client-side A/B testing means that changes to your A/B test take place in the visitor's browser. Using a client-side A/B testing tool, setting up a simple test is not very complicated (although familiarity with HTML, CSS and JS is helpful). However, the relative ease of use of client-side also has drawbacks. The options in terms of testing are mainly limited to design. Think: colors, text, layout and elements. For some organizations, this is fine. There are countless testing ideas you can run client-side, but at some point you want more options. This is where server-side A/B testing comes in.

What is server-side testing?

Server-side A/B testing is a form of experimentation in which the variation of a test is displayed directly on the Web server rather than in the user's browser. With server-side testing, developers, using code, can live the A/B tests directly on the servers. Because server-side implementation is more direct, it allows for much more sophisticated A/B testing. However, this means that the person setting up the test must be experienced in back-end coding languages, such as PHP or Node.js.

What are the benefits of server-side testing?

With server-side testing, it is possible to run complicated A/B tests. Think about product (features), algorithms and sequence steps in the checkout. Other reasons to choose a server-side solution are:

• A/B testing is of higher quality because you avoid flickering (when an original page is briefly displayed before the B variant appears), do not use a WYSIWYG editor (this often leads to bad code) and can implement winners immediately because they are already ready on the server.
• Is often preferred by development because server-side A/B testing fits better within the website code and the developers are more in control.
• More teams can test independently. For example, product teams can self-validate their new developments before putting them live.
• It meets security and privacy requirements. With client-side testing, an external script from your tool has access to your website and data, this poses risks. Especially if it is placed behind a login or in places where client data is available. 
• Safari's ITP speeds up the deletion of your first-party cookies (1-7 days). This affects the quality of your data and can cause visitors to see a different variant of your test in different sessions. If a visitor comes back to your website after his cookie has been deleted, then this visitor is seen as new. This creates a new distribution in your A/B test, so you can't be sure that the user has seen the same variant each time. And because the user is seen as new, the number of unique users increases significantly. Depending on your visitors and duration of your test, this can have a significant impact on your data and your results. First-party cookies can be set both from the server and from the client (through javascript). ITP focuses specifically on the use of client-side cookies. A/B testing and analytics tools use these client-side first-party cookies because they ensure that a user is recognized across sessions and that a user continues to see the same variation of an A/B test over and over again. By setting cookies server-side you can prevent your cookies from being deleted and looking forward, this is a good reason to switch. The next ITP update ensures that this will apply to all browsers on an iOS device and thus cookies will be deleted after seven days (and with a ‘decorated link’ after just one day).
• The final benefit is cost savings, if you create the server-side solution yourself and don't use an external tool for it (more on this later). Annual costs for a client-side tool can be high, especially on high-traffic websites. Whereas server-side testing sometimes has the perception of being expensive (because the solution needs to be built and maintained) this does not always turn out to be the case in practice. It does depend on the type of solution you choose (both client- and server-side), but it does not automatically mean that a server-side solution is more expensive.

What are the challenges of server-side A/B testing?

If server-side A/B testing had only benefits then everyone would be doing it. Suppose you want to switch to server-side, there are a number of challenges to consider:

• Sufficient development capacity is needed in the product teams. With server-side testing, the development of the experiments lies with the internal teams and no longer with a front-end developer who is also on the CRO team. The developers now not only implement the winning experiments, but they have to develop every test. If there are other (internal) priorities, your testing program is delayed because development has less time to build A/B tests.
• The right mindset in the product teams. The scrum method is focused on executing and completing a project. Of course, this includes looking at the potential value. If this project is validated through an experiment, this may mean that certain deliverables are not put live if they are a loser, even though they have already been built. This impacts the ‘mindset’ of product teams and can cause lower motivation. Something that a lot of time and effort was put into during a sprint suddenly doesn't go live anyway.
• Development capacity for creating and continuing to develop your tool. Even if your first variant of your self-developed server-side tool is live, it does not mean it is finished. Development capacity is needed to continue developing and maintaining your tool. There are costs associated with this as well.
• Standardize the process. If you're just starting out with server-side testing, the necessary time and energy goes into getting all teams to work this way. Of course, you don't have to transition all at once and this can be a gradual process.
• Server-side data is generally more reliable than the data in a platform such as Google Analytics. This just needs to be built properly, which presents another challenge. Initially, the analyses can simply still be made in the analytics tool before it is integrated into the server-side solution.

Solutions

If you want to test server-side, there are actually two solutions:

1. Creating your own server-side solution.
2. You can use an external testing tool to implement your tests (as in client-side). Most testing tools (except Google) have or are developing a server-side solution.

Suppose you are going to create your own server-side solution, keep in mind the following points:

• Start small to combat complexity (experiment with experimentation).
• Tips to get developers on board and keep them motivated:
  • A/B testing should become a starting point, not a question.
  • Experimentation must go along with the rhythm of scrum teams.
  • Share unexpected learnings from A/B testing.
  • Teams can be given an outcome KPI (more conversions), rather than just a delivery KPI (five A/B tests per month).
  • Show that only a delivery KPI leads to positive impact in a limited number of cases.
• Server-side measurement makes bucketing possible and data more reliable. As indicated above, server-side measurement does present another challenge. It has to be built (well). In this case: start small. For example, start by measuring only the number of visitors and conversions on the most important metric.
• There is an assumption that it is very difficult to build and maintain a server-side solution yourself. Following a meetup with companies that have built their own solution, this turns out to be not that difficult. Certainly the first version in which visitors have to be randomly distributed is not too complicated. An important learning is: just start building it and don't keep discussing it forever.
• When teams lose speed and control, a hybrid solution can be chosen. Here you make tests concerning design via your test tool and load them on the client-side, and do complicated tests via your server-side solution. This is also highly recommended if you are just starting out with server-side A/B testing.

For us, the advantages certainly outweigh the disadvantages of (partly) server-side testing. At some point you will run into the limited possibilities and lower quality of client-side testing. Good luck and keep testing 🙂 

Want to know more?

We'd love to think with you! Need help or advice? If so, please contact Ruben de Boer.

Want to receive the latest news and articles about our field every month? Sign up for our newsletter!

Ruben de Boer and Tom van den Berg also discuss server-side experimentation in an English-language podcast for CRO.CAFE. Listen to the podcast here.