Regression test suite prioritization techniques reorder a test suite with the goal of ensuring that the reorganized test suite finds faults faster than the initial ordering. It is challenging to empirically evaluate the effectiveness of a new test case arrangement because existing metrics (i) require fault seeding or (ii) ignore test case costs. This paper presents a coverage effectiveness (CE) metric that (i) obviates the need to seed faults into the program under test and (ii) incorporates available data about test case execution times. A test suite is awarded a high CE value when it quickly covers the test requirements. It is possible to calculate coverage effectiveness regardless of the coverage criterion that is chosen to evaluate test case quality. The availability of an open source CE calculator enables future case studies and controlled experiments to use coverage effectiveness when evaluating different approaches to test suite prioritization.
Kapfhammer, G. M., & Soffa, M. L. (2007). Using coverage effectiveness to evaluate test suite prioritizations. Proceedings of the International Workshop on Empirical Assessment of Software Engineering Languages and Technologies.
Want to cite this paper? Look in the BiBTeX file of gkapfham/research-bibliography for the key "Kapfhammer2007".