Mind the Gap: What Working With Developers on Fuzz Tests Taught Us About Coverage Gaps

Carolin Brandt*, Marco Castelluccio, Christian Holler, Jason Kratzer, Andy Zaidman, Alberto Bacchelli

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

1 Citation (Scopus)
6 Downloads (Pure)

Abstract

Can fuzzers generate partial tests that developers find useful enough to complete into functional tests (e.g., by adding assertions)? To address this question, we develop a prototype within the Mozilla ecosystem and open 13 bug reports proposing partial generated tests for currently uncovered code. We found that the majority of the reactions focus on whether the targeted coverage gap is actually worth testing. To investigate further which coverage gaps developers find relevant to close, we design an automated filter to exclude irrelevant coverage gaps before generating tests. From conversations with 13 developers about whether the remaining coverage gaps are worth closing when a partially generated test is available, we learn that the filtering indeed removes clearly non-test-worthy gaps. The developers propose a variety of additional strategies to address the coverage gaps and how to make fuzz tests and reports more useful for developers.

Original languageEnglish
Title of host publicationProceedings - 2024 ACM/IEEE 44th International Conference on Software Engineering
Subtitle of host publicationNew Ideas and Emerging Results, ICSE-SEIP 2024
PublisherAssociation for Computing Machinery (ACM)
Pages157-167
Number of pages11
ISBN (Electronic)9798400705007
DOIs
Publication statusPublished - 2024
Event2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-SEIP 2024 - Lisbon, Portugal
Duration: 14 Apr 202420 Apr 2024

Publication series

NameACM International Conference Proceeding Series

Conference

Conference2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-SEIP 2024
Country/TerritoryPortugal
CityLisbon
Period14/04/2420/04/24

Fingerprint

Dive into the research topics of 'Mind the Gap: What Working With Developers on Fuzz Tests Taught Us About Coverage Gaps'. Together they form a unique fingerprint.

Cite this