A survey of flaky tests

Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil
Transactions on Software Engineering and Methodology 31:1, 2022

Abstract

Tests that fail inconsistently, without changes to the code under test, are described as flaky. Flaky tests do not give a clear indication of the presence of software bugs, and thus limit the reliability of the test suites that contain them. A recent survey of software developers found that 59% claimed to deal with flaky tests on a monthly, weekly or daily basis. As well as being detrimental to developers, flaky tests have also been shown to limit the applicability of useful techniques in software testing research. In general, one can think of flaky tests as being a threat to the validity of any methodology that assumes the outcome of a test only depends on the source code it covers. In this paper, we systematically survey the body of literature relevant to flaky test research, amounting to 76 papers. We split our analysis into four parts, addressing the causes of flaky tests, their costs and consequences, detection strategies, and approaches for their mitigation and repair. Our findings and their implications have consequences for how the software testing community deals with test flakiness, pertinent to practitioners and of interest to those wanting to familiarize themselves with the research area.

Resources

Reference

Parry, O., Kapfhammer, G. M., Hilton, M., & McMinn, P. (2022). A survey of flaky tests. Transactions on Software Engineering and Methodology, 31(1).

Citing

Want to cite this paper? Look in the BiBTeX file of gkapfham/research-bibliography for the key "Parry2022".

Return to the List of Papers

Please support my work!

View the source.