Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

77 Citations (Scopus)
52 Downloads (Pure)


AbContinuous Integration (CI) has become a best practice of modern software development. Yet, at present, we have a shortfall of insight into the testing practices that are common in CI-based software development. In particular, we
seek quantifiable evidence on how central testing is to the CI process, how strongly the project language influences testing, whether different integration environments are valuable and if testing on the CI can serve as a surrogate to local testing in the IDE. In an analysis of 2,640,825 Java and Ruby builds on
TRAVIS CI, we find that testing is the single most important reason why builds fail. Moreover, the programming language has a strong influence on both the number of executed tests, their run time, and proneness to fail. The use of multiple integration environments leads to 10% more failures being caught at build time. However, testing on TRAVIS CI does not seem an adequate
surrogate for running tests locally in the IDE. To further research on TRAVIS CI with GITHUB, we introduce TRAVISTORRENT.
Original languageEnglish
Title of host publicationProceedings - 2017 IEEE/ACM 14th International Conference on Mining Software Repositories, MSR 2017
Place of PublicationLos Alamitos, CA
Number of pages12
ISBN (Electronic)978-1-5386-1544-7
Publication statusPublished - 2017
EventMSR 2017: 14th International Conference on Mining Software Repositories - Buenos Aires, Argentina
Duration: 20 May 201721 May 2017
Conference number: 14


ConferenceMSR 2017
Abbreviated titleMSR
CityBuenos Aires
Internet address


  • Testing
  • Software
  • Java
  • Tools
  • Best practices
  • Programming


Dive into the research topics of 'Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub'. Together they form a unique fingerprint.

Cite this