Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage Checks

Alexander Sterk, Mairieli Wessel, Eli Hooten, Andy Zaidman*

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

Abstract

Many modern code coverage tools track and report code coverage data generated from running tests during continuous integration. They report code coverage data through a variety of channels, including email, Slack, Mattermost, or through the web interface of social coding platforms such as GitHub. In fact, this ensemble of tools can be configured in such a way that the software engineer gets a failing status check when code coverage drops below a certain threshold. In this study, we broadly investigate the opinions and experience with code coverage tools through a survey among 279 software engineers whose projects use the Codecov coverage tool and bot. In particular, we are investigating why software engineers would ignore a failing status check caused by drop in code coverage. We observe that >80% of software engineers-at least sometimes-ignore these failing status checks, and we get insights into the main reasons why software engineers ignore these checks.

Original languageEnglish
Title of host publicationProceedings - 2024 IEEE/ACM International Conference on Automation of Software Test, AST 2024
PublisherAssociation for Computing Machinery (ACM)
Pages12-22
Number of pages11
ISBN (Electronic)9798400705885
DOIs
Publication statusPublished - 2024
Event5th ACM/IEEE International Conference on Automation of Software Test, AST 2024, co-located with the 46th International Conference on Software Engineering, ICSE 2024 - Lisbon, Portugal
Duration: 15 Apr 202416 Apr 2024

Publication series

NameProceedings - 2024 IEEE/ACM International Conference on Automation of Software Test, AST 2024

Conference

Conference5th ACM/IEEE International Conference on Automation of Software Test, AST 2024, co-located with the 46th International Conference on Software Engineering, ICSE 2024
Country/TerritoryPortugal
CityLisbon
Period15/04/2416/04/24

Keywords

  • code coverage
  • coverage checks
  • software testing

Fingerprint

Dive into the research topics of 'Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage Checks'. Together they form a unique fingerprint.

Cite this