Challenges in Applying Continuous Experimentation: A Practitioners' Perspective

Kevin Anderson, Denise Visser, Jan-Willem Mannen, Yuxiang Jiang, Arie van Deursen

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

55 Downloads (Pure)


Background: Applying Continuous Experimentation on a large scale is not easily achieved. Although the evolution within large tech organisations is well understood, we still lack a good understanding of how to transition a company towards applying more experiments. Objective: This study investigates how practitioners define, value and apply experimentation, the blockers they experience and what to do to solve these. Method: We interviewed and surveyed over one hundred practitioners with regards to experimentation perspectives, from a large financial services and e-commerce organization, based in the Netherlands. Results: Many practitioners have different perspectives on experimentation. The value is well understood. We have learned that the practitioners are blocked by a lack of priority, experience and well functioning tooling. Challenges also arise around dependencies between teams and evaluating experiments with the correct metrics. Conclusions: Organisation leaders need to start asking for experiment results and investing in infrastructure and processes to actually enable teams to execute experiments and show the value of their work in terms of value for customers and business.
Original languageEnglish
Title of host publicationProceedings - 2022 ACM/IEEE 44th International Conference on Software Engineering
Subtitle of host publicationSoftware Engineering in Practice, ICSE-SEIP 2022
Number of pages8
ISBN (Electronic)978-1-6654-9590-5
ISBN (Print)978-1-6654-9591-2
Publication statusPublished - 2022
EventThe 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) - Pittsburgh, United States
Duration: 22 May 202224 May 2022
Conference number: 44th

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257


ConferenceThe 2022 IEEE/ACM 44th International Conference on Software Engineering
Country/TerritoryUnited States


  • Continuous experimentation
  • Online controlled experiments
  • A/B testing
  • Empirical software engineering
  • ING


Dive into the research topics of 'Challenges in Applying Continuous Experimentation: A Practitioners' Perspective'. Together they form a unique fingerprint.

Cite this