Flake it ‘till you make it: Using automated repair to induce and fix latent test flakiness

Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil
Proceedings of the 1st International Workshop on Automated Program Repair, 2020

Abstract

Since flaky tests pass or fail nondeterministically, without any code changes, they are an unreliable indicator of program quality. Developers may quarantine or delete flaky tests because it is often too time consuming to repair them. Yet, since decommissioning too many tests may ultimately degrade a test suite’s effectiveness, developers may eventually want to fix them, a process that is challenging because the nondeterminism may have been introduced previously. We contend that the best time to discover and repair a flaky test is when a developer first creates and best understands it. We refer to tests that are not currently flaky, but that could become so, as having latent flakiness. We further argue that efforts to expose and repair latent flakiness are valuable in ensuring the future-reliability of the test suite, and that the testing cost is greater if latent flakiness is left to manifest itself later. Using concrete examples from a real-world program, this paper posits that automated program repair techniques will prove useful for surfacing latent flakiness.

Resources

Reference

Parry, O., Kapfhammer, G. M., Hilton, M., & McMinn, P. (2020). Flake it ‘till you make it: Using automated repair to induce and fix latent test flakiness. Proceedings of the 1st International Workshop on Automated Program Repair.

Citing

Want to cite this paper? Look in the BiBTeX file of gkapfham/research-bibliography for the key "Parry2020".

Return to the List of Papers

Please support my work!

View the source.