A benchmark-based evaluation of search-based crash reproduction

Mozhan Soltani, Pouria Derakhshanfar, Xavier Devroey, Arie van Deursen

Research output: Contribution to journalArticleScientificpeer-review

22 Citations (Scopus)
222 Downloads (Pure)


Crash reproduction approaches help developers during debugging by generating a test case that reproduces a given crash. Several solutions have been proposed to automate this task. However, the proposed solutions have been evaluated on a limited number of projects, making comparison difficult. In this paper, we enhance this line of research by proposing JCrashPack, an extensible benchmark for Java crash reproduction, together with ExRunner, a tool to simply and systematically run evaluations. JCrashPack contains 200 stack traces from various Java projects, including industrial open source ones, on which we run an extensive evaluation of EvoCrash, the state-of-the-art tool for search-based crash reproduction. EvoCrash successfully reproduced 43% of the crashes. Furthermore, we observed that reproducing NullPointerException, IllegalArgumentException, and IllegalStateException is relatively easier than reproducing ClassCastException, ArrayIndexOutOfBoundsException and StringIndexOutOfBoundsException. Our results include a detailed manual analysis of EvoCrash outputs, from which we derive 14 current challenges for crash reproduction, among which the generation of input data and the handling of abstract and anonymous classes are the most frequents. Finally, based on those challenges, we discuss future research directions for search-based crash reproduction for Java.
Original languageEnglish
Pages (from-to)96–138
Number of pages43
JournalEmpirical Software Engineering
Issue number1
Publication statusPublished - Jan 2020


  • Crash reproduction
  • Empirical evaluation
  • Search-based software testing


Dive into the research topics of 'A benchmark-based evaluation of search-based crash reproduction'. Together they form a unique fingerprint.

Cite this