Experimentation (A/B testing) on the server is a topic of increasing interest. When you decide to switch to server-side testing (instead of client-side), you are faced with an important choice: build or buy.

Building the basic functions of your server-side testing tool is often easier than most people think. And in most cases, it's also cheaper than buying an existing solution.

In this article, I describe the benefits of running server-side experiments and a chronological roadmap of ten features you can build into your server-side testing tool.

Before you start this, you should have a basic understanding of the difference between client-side and server-side experimentation. When you type a URL into your browser, it sends a request to a server. This server sends data back to your browser, causing you to see the Web site. With client-side experiments, the changes you make are loaded into the browser. With server-side, the changes are loaded on the server before being sent to your visitor's browser. Want to know more about server-side testing? Then read this blog.

Why run experiments on the server?

There are several advantages when you run experiments on your server:

• It is possible to perform complicated A/B testing. These include testing product (features), algorithms and the sequence of steps at checkout.
• There is no flickering. This occurs when the original page is briefly displayed before the B variant appears. Also called Flash or Original Content (FOOC) called.
• It is often preferred by development, because server-side A/B testing is a better fit for website code.
• Winning experiments can be implemented immediately.
• It uses server-side cookies. Since more and more browsers delete client-side cookies after a few days, this makes your test data more reliable.

There are also some drawbacks:

• It is more difficult to set up experiments, since you generally have to use your website's code language. Although I have also seen solutions with the ability to set up experiments with Javascript and CSS (as in client-side tools).
• There must be sufficient development capacity to conduct experiments.
• If you are using an existing solution (tools like Optimizely or ABTasty have server-side solutions), this can get quite expensive.
• When you build the tool, you need developers to maintain and update the tool.

In my experience, building the tool saves you money over buying the tool.

Building the 10 functions for your server-side test tool

When you decide to build the server-side testing tool, it is recommended that you implement the following features chronologically. It is essential to start small to reduce complexity.

1. The coinflip

The coinflip splits website visitors so that the control and variant(s) are shown to the right visitors. This is generally relatively easy to build. The targeting might be a little more complicated.

The same principle applies here; start small. For example, first with targeting by device.

2. Stop / start button

A stop and start button allows more people to start and stop an experiment. Stopping an experiment can be important if the variation performs much worse or causes errors.

3. User Interface

Having a user interface makes your tool more user-friendly, allowing more colleagues to use it. This helps democratize experiments, allowing more product teams to run their experiments and validate their ideas.

Somewhere between function 1 and 3

Somewhere between feature 1 and 3, you should create a QA (quality assurance) or sample link. This helps other people verify that the variant is displayed and performs as it should, before the experiment goes live.

At some point, you'll also want to build a start and end date function into the tool so you can schedule experiments more easily.

4. Server-side data: key KPI

Server-side data are generally more reliable than data on a platform such as Google Analytics (client-side). Setting up server-side measurements can be challenging. One solution, again, is to start small. For example, start by measuring only the number of visitors and conversions of your most important KPI.

5. Server-side data: different KPIs.

The next step would be to include metrics for multiple KPIs.

6. Guardrail metrics and security controls.

With your data available in the tool, you can set guardrail metrics. Guardrail metrics are crucial metrics for your website that you use to set a threshold before experimenting. When the numbers exceed the threshold, the experiment should be terminated or not declared a winner. This can happen when it suddenly becomes apparent that your changes are having an unexpected negative impact on your business. Page speed is a common guardrail metric, as is browser-specific performance. You can use these guardrail metrics as security checks.

7. Advanced segmentation

Because a lot of data is available in the tool, you can now set up advanced segmentations, allowing personalization experiments.

In many situations, this feature can be built in earlier if you have a data management platform that can connect to your tool.

8. Automate calculations

The more time you save calculating your results, the more experiments you can run. When you automate the calculations, including the significance calculation, your analysts need less time to analyze the results of the experiments.

9. Reporting and documentation

The next step could be automated reporting and documentation.

10. Automated experiments

The final step is automated experimentation. Here, Scrum teams and developers can no longer put updates live for all visitors. So every release will automatically be an experiment. This way, the organization is sure that only winners will be put fully live, resulting in growth.

Experimenting on the server is the future

This article gave a brief overview of the features to consider in your server-side testing tool. The future of online experiments lies in server-side testing solutions. Not only can you set up more complex experiments, it also results in more reliable experiments because of privacy rules, server-side tracking and no flicker effect. When building or buying, consider your long-term goals and find the best solution.

If your company does not have the experience to build such tools, contact others in the market. As they will probably tell you, coinflip is relatively easy to build. Therefore, get started, start small and work from there. Good luck!

Want to stay up to date on trends, developments and events in the CRO world? Sign up for our newsletter!