Challenges in Applying Continuous Experimentation: A Practitioners' Perspective

Kevin Anderson, Denise Visser, Jan-Willem Mannen, Yuxiang Jiang, Arie van Deursen

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

86 Downloads (Pure)

Abstract

Background: Applying Continuous Experimentation on a large scale is not easily achieved. Although the evolution within large tech organisations is well understood, we still lack a good understanding of how to transition a company towards applying more experiments. Objective: This study investigates how practitioners define, value and apply experimentation, the blockers they experience and what to do to solve these. Method: We interviewed and surveyed over one hundred practitioners with regards to experimentation perspectives, from a large financial services and e-commerce organization, based in the Netherlands. Results: Many practitioners have different perspectives on experimentation. The value is well understood. We have learned that the practitioners are blocked by a lack of priority, experience and well functioning tooling. Challenges also arise around dependencies between teams and evaluating experiments with the correct metrics. Conclusions: Organisation leaders need to start asking for experiment results and investing in infrastructure and processes to actually enable teams to execute experiments and show the value of their work in terms of value for customers and business.
Original languageEnglish
Title of host publicationProceedings - 2022 ACM/IEEE 44th International Conference on Software Engineering
Subtitle of host publicationSoftware Engineering in Practice, ICSE-SEIP 2022
PublisherIEEE
Pages107-114
Number of pages8
ISBN (Electronic)978-1-6654-9590-5
ISBN (Print)978-1-6654-9591-2
DOIs
Publication statusPublished - 2022
Event44th ACM/IEEE International Conference on Software Engineering, ICSE 2022: Software Engineering in Practice (ICSE-SEIP) - Pittsburgh, United States
Duration: 22 May 202227 May 2022
Conference number: 44th

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257

Conference

Conference44th ACM/IEEE International Conference on Software Engineering, ICSE 2022
Abbreviated title ICSE 2022
Country/TerritoryUnited States
CityPittsburgh
Period22/05/2227/05/22

Keywords

  • Continuous experimentation
  • Online controlled experiments
  • A/B testing
  • Empirical software engineering
  • ING
  • bol.com

Fingerprint

Dive into the research topics of 'Challenges in Applying Continuous Experimentation: A Practitioners' Perspective'. Together they form a unique fingerprint.

Cite this