Growing from a company with data to a data-driven company is not a step but an evolution. Aleksander Fabijan studied this at Microsoft and formulated four stages to achieve continuous experimentation. A summary of this model by Ton Wesseling:

Aleksander Fabijan elaborates on his model in a keynote and workshop at Conversion Hotel 2018, From November 16 to 18 on Texel

Crawl

Organizationally, technically and operationally, everything is still in its proverbial infancy. There is no experimentation team yet; it is the data scientist who performs a number of experiments independently. From manual implementation to manual analysis. Nor are these experiments set up through a solid experiment platform. And the assessment of the result of the experiment is done based on a metric that is still to be determined for each experiment. Whereby one often runs into data quality problems and product development choices within the company are still primarily made based on historical behavioral data. Experiments are purely still showcases.

Walk

A central experimentation team has been established. However, product teams themselves do not yet have far-reaching knowledge about experiments, but are already developing the variants to be tested themselves. There is a stable external or self-developed platform for experiments, with the random distribution well in place. And A/A tests and solid power calculations are used before experiments are started. A fixed set of debug, success and data quality metrics are used as evaluation criteria. Experiments take place on features (e.g., form design or Web site performance). Experiment outcomes begin to impact product teams' release planning.

Run

Product teams now have responsibility for their own experiments. From design to monitoring to decisions to stop or scale up. Data scientists from the central experiments team are part of the product teams, so they can become domain experts and grow experiments within their product teams. The experiment platform used supports scaling in experiments through alert functions, control of carry-over effects and experiment iteration support. Data quality is fully in place with a validated assessment criterion and there is automated analysis of experiments. Experimentation impacts product quality and product complexity by testing almost every new feature.

Fly

Each product team has its own experimenter who together form a strong unit within the organization. This central experiment team acts as a center of excellence within the organization. All changes, even bug fixes, are started as experiments. An intelligent platform is used with real-time checks for interaction effects, and harmful experiments are automatically stopped. The experiment rating crit eria are stable with only annual changes. These metrics are also used to drive the organization. Experimentation is in the DNA of the organization. Everyone thinks from experimentation in every development. From the bottom to the top of the organization.

Ton Wesseling, on behalf of Online Dialogue, fills the recurring conversion column each month in Emerce magazine: the magazine about “the next step in E-business.” This month he wrote about: The evolution of continuous experimentation.

Click on the image to enlarge it, or download the pdf here.

Emerce Conversion Feature September 2018