Change-based test selection: An empirical evaluation

Quinten David Soetens, Serge Demeyer, Andy Zaidman, Javier Perez

Research output: Contribution to journalArticleScientificpeer-review

22 Citations (Scopus)
113 Downloads (Pure)


Regression test selection (i.e., selecting a subset of a given regression test suite) is a problem that has been studied intensely over the last decade. However, with the increasing popularity of developer tests as the driver of the test process, more fine-grained solutions that work well within the context of the Integrated Development Environment (IDE) are in order. Consequently, we created two variants of a test selection heuristic which exploit fine-grained changes recorded during actual development inside the IDE. One variant only considers static binding of method invocations while the other variant takes dynamic binding into account. This paper investigates the tradeoffs between these two variants in terms of the reduction (i.e., How many tests could we omit from the test suite, and how much did we gain in runtime execution?) as well as the fault detection ability of the reduced test suite (i.e., Were tests omitted erroneously?). We used our approach on three distinct cases, two open source cases —Cruisecontrol and PMD— and one industrial case — Historia. Our results show that only considering static binding reduces the test suite significantly but occasionally omits a relevant test; considering dynamic binding rarely misses a test yet often boils down to running the complete test suite. Nevertheless, our analysis provides indications on when a given variant is more appropriate.
Original languageEnglish
Pages (from-to)1990-2032
Number of pages43
JournalEmpirical Software Engineering
Issue number5
Publication statusPublished - 11 Nov 2015


  • Fine grained changes
  • Test selection
  • Developer tests
  • Dynamic binding


Dive into the research topics of 'Change-based test selection: An empirical evaluation'. Together they form a unique fingerprint.

Cite this