Automated Sample Ratio Mismatch (SRM) Detection and Analysis

Lukas Vermeer, Kevin Anderson, Mauricio Acebal

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

15 Downloads (Pure)

Abstract

Background: Sample Ratio Mismatch (SRM) checks can help detect data quality issues in online experimentation [3]. Not all experimentation platforms provide these checks as part of their solution. Users of these platforms must therefore manually check for SRM, or rely on additional processes—such as checklists [2]—or automation. Objective: To ensure reliable and early detection of SRM, we wanted to automate the detection and analysis of SRM in experiments running on third-party experimentation platforms. Method: A set of Looker dashboards were built to facilitate self-serve SRM detection and root cause analysis. In addition, we added email and chat based alerting to pro-actively inform experimenters of SRM and guide them towards these dashboards when needed. Results: Several cases of SRM have been detected and experimenters have been warned. Bad decisions based on flawed data were avoided. We provide one such example as an illustration. Conclusions: SRM checks are relatively straightforward to automate and can be useful for data quality monitoring even for companies who rely on third-party experimentation platforms. Pro-active alerting—rather than passive reporting—can reduce time to detection and help non-experts avoid making decisions based on biased data.
Original languageEnglish
Title of host publicationProceedings of the ACM International Conference on Evaluation and Assessment in Software Engineering, EASE 2022
Subtitle of host publicationThe International Conference on Evaluation and Assessment in Software Engineering 2022
EditorsM. Staron, C. Berger, J. Simmonds, R. Prikladnicki
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Pages268–269
Number of pages2
ISBN (Electronic)9781450396134
ISBN (Print)978-1-4503-9613-4
DOIs
Publication statusPublished - 2022
EventEASE 2022: The International Conference on Evaluation and Assessment in Software Engineering 2022 - Gothenburg, Sweden
Duration: 13 Jun 202215 Jun 2022

Publication series

NameACM International Conference Proceeding Series

Conference

ConferenceEASE 2022
Country/TerritorySweden
CityGothenburg
Period13/06/2215/06/22

Keywords

  • Infrastructure
  • Data Quality
  • Trustworthiness
  • A/B Testing
  • Sample Ratio Mismatch
  • SRM
  • Online Controlled Experimentation

Fingerprint

Dive into the research topics of 'Automated Sample Ratio Mismatch (SRM) Detection and Analysis'. Together they form a unique fingerprint.

Cite this