Scrum & “Evidence Based” experimentation: how these practices can reinforce each other.

Companies want to make customers happy and communicate the value of their services, products and experiences in the right way. In the world of digital marketing, the response to customers‘ wishes and needs is becoming smarter, faster and easier. The focus here is both on ’quickly putting live‘ and the ’continuous optimization" of (new) products, services and digital channels.

“Attention Makes Everything More Beautiful,” “Anything for a Smile,” or “A State of Happiness.”.

From Online Dialogue, we see companies increasingly focusing on continuous optimization (CRO). CRO is based on the belief that growth or mitigating risk occurs through quick adjustments or improvements that are substantiated with results (data and behavioral knowledge) from experiments. You have an idea based on data or other insights, you test this idea, you learn from the outcome, adjust it, and you grow in conversion and knowledge. In this way more and more knowledge is gained about which communication, proposition, product or service works, but in practice these insights are often not yet included in the development of new digital services and products within the company. So time to unite innovation and optimization!

Scrum & Optimization

One of the most well-known and popular practices used to establish something new or make an existing service or product better step by step is the Scrum method. This working method is increasingly being adopted by different departments of small and large organizations. Despite the many similarities and valuable complementary knowledge and applications between Scrum and CRO, in practice I see that these two ways of working are often still used separately from each other. In this article I describe the two ways of working and highlight the similarities (and differences). I also offer insights and tools to unify the two ways of working and to organize them differently, with of course much room for other (complementary) applications and ways of organizing.

The agreements

The Sprint & the CRO Process.

The heart of scrum is the Sprint. The Sprint is “a period of one month or less in which a usable and potentially releasable product is delivered.” We apply a similar structure to the Conversion Optimization process via the FACT & ACT model. This model aims to arrive at a list of substantiated ideas in a short time (FACT) after thorough research and validate them as quickly as possible through A/B testing (ACT). Especially when validating ideas, it is important to create new experiments and put them live at a fast pace.

In both Scrum and CRO, the process focuses on iteration. In CRO, you iteratively learn to understand visitor behavior better and better. With scrum, you iteratively arrive at a usable product. Both approaches deliver a visible and measurable result after a short period of time. Indispensable in such a process are transparency, control and room to adjust.

The team

The basis of a successful project or program both in CRO and within the scrum way of working are the capabilities and organization of the self-managing team. In the CRO world, such a team ideally consists of: a project/program lead, an analyst, a copywriter, a behavioral scientist, a UX designer and a developer. A multidisciplinary team working together to optimize a component of a site, funnel or the entire online dialogue.

Also according to the Scrum framework, a self-managing team consists of multiple disciplines. In the case of Scrum, this means: A product owner (responsible for managing the Product Backlog), a scrum master: (a (spiritual) leader and coach for the Scrum Team) and the development team (consisting of the professionals who deliver the software or functionalities). If you work exactly according to scrum rules then the team members all lose their job titles and simply become “developers,” contributing their strengths and weaknesses to the team to achieve the goals.

Transparency through various meetings

To improve your products and/or services, but especially your own work, you need transparency. You need to be aware of the work of your teammates and of all developments within the project. To achieve this, both Scrum and the CRO process use moments to share new information and developments with each other. I have put them side by side for convenience.

The difference between Scrum and CRO

Besides the many similarities, there is also a clear difference between Scrum and CRO. For example, Scrum involves working to deliver a usable product through several Sprints and thus has a temporary(er) nature, because a new program/project can always be started or another issue is prioritized. CRO, on the other hand, is not temporary. It is a continuous process in which interesting and applicable insights are gathered that are valuable in the short term, but mainly bear fruit in the long term.

How to move faster?

The power of optimization and scrum is to arrive quickly and smarter at functionalities, products and services that are valuable to visitors or customers. In both approaches, the team itself is able to choose what is created, how it is done and how it leads to tangible products. But how do you make both ways of working together grow your organization? How can they be woven together in practice?

Decisions & the backlog

In the article “How A/B Testing at LinkedIn, Wealthfront and eBay Made Me a Better Manager” states Elliot Shmukler, former product lead at Linkedin, Ebay and Vice President at Instacart, on how he looks at A/B testing and decision making. He is referring to Jeff Bezos who, back in 1998, indicated that there are 2 types of decisions:

Type 1 decisions:“Some decisions are consequential and irreversible or nearly irreversible - one-way doors - and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation. If you walk through and don't like what you see on the other side, you can't get back to where you were before.”

Type 2 decisions: “But most decisions aren't like that - they are changeable, reversible - they're two-way doors. If you've made a suboptimal Type 2 decision, you don't have to live with the consequences for that long. You can reopen the door and go back through. Type 2 decisions can and should be made quickly by high judgment individuals or small groups.”

As organizations get larger, there seems to be a tendency to use the heavy-weight Type 1 decision-making process on most decisions, including many Type 2 decisions. The end result of this is slowness, unthoughtful risk aversion, failure to experiment sufficiently, and consequently diminished invention.

Shmukler hereby indicates that: “In the traditional sense, A/B testing is about having at least two versions of the product live: an A version, typically the original implementation or control, and a B version, which you think might be better,” says Shmukler. “Thus when A/B testing is applied to Type 2 management decisions, it's very easy to walk back through the door as Bezos suggests by simply turning off B and returning to A.”

A/B testing helps to make (and possibly reverse) decisions quickly and better, delivering value to customers. This is a key deliverable for Product owners, so I think they play an important role in weaving CRO and Scrum together.

Making smaller, but testing

In an organization built according to Scrum, the Product owner and the team have the mandate to make testing a structural part of the Sprint. However, time and capacity must be reserved on the backlog. Something that we unfortunately see happen too rarely in practice, causing CRO work to be delayed rather than accelerated. By including CRO in the backlog and slightly adjusting the goals for each Sprint, you free up space to not only renew your product but also to test it, thus ensuring that your renewal is actually the right one.

Product development vs. product improvement

Testing your product, proposition or communication can be done with new products (product development) as well as with existing products (product improvement). In practice, we see that especially in product development sprints, knowledge of customer behavior in the organization is not taken into account enough. Test teams should meet regularly with the product development teams to exchange knowledge and share results. For this, the organization will need to set up a structure where the teams sit together more often, for example, by appointing a separate coordinating Customer Knowledge Centre to oversee, understand or execute testing.

Ever faster learning, adaptation and growth are essential in constantly changing environments. Methods such as Scrum and CRO, among others, will help organizations achieve this. Now it is only up to scrum teams and test teams to combine the complementary sides of both approaches so that they do not hold each other back but rather enhance each other.

This article was published on April 3 at Emerce