Introducing Major for the mutation analysis of the tests for Java programs

post
mutation testing
software tool
Have JUnit tests? Use Major to assess their quality!
Author

Gregory M. Kapfhammer

Published

2014

If you have a JUnit test suite, it is important that you ask the question “how good are my test cases?” If you have a low quality test suite, then you may not be doing a good job at establishing a confidence in the correctness of the program under test—and there also could be defects lurking in the application!

One strategy for assessing the quality of your test suite is to run coverage analysis with a tool like JaCoCo. Since a test case cannot find a defect unless it first executes it, the idea is that you can judge a test suite to be good if it adequately covers the code in the program under test. However, coverage analysis does not help a tester to know with certainty that the tests are design in such a way that they are likely to find defects. As an alternative to, or in addition to coverage analysis, mutation testing inserts simple faults into the program under test to determine whether or not the tests can find them. Intuitively, a test case that can find many of these faults is a good one.

While mutation analysis is simple and useful, historically, it has been difficult to apply to large-scale programs because it was computationally expensive. This all changed with the design, implementation, and release of Major, a compiler-integrated mutation testing tool for the Java programming language. You can learn more about Major by reading (Just, Kapfhammer, and Schweiggert 2011b) and (Just, Kapfhammer, and Schweiggert 2011a) . More details about how Major makes mutation testing scalable through use of non-redundant mutation operators and test suite prioritization is available in (Just, Kapfhammer, and Schweiggert 2012) . Starting to use the tool is as easy as changing the Java compiler that you use at build time, as demonstrated by the following line from Major’s online documentation.

$ javac -XMutator:ALL MyFile.java
# Generated Mutants: 190 (96 ms)

The creator of Major, Rene Just, took great care to ensure that it is fast, allowing mutation testing to be applied to large real-world programs. Major version 2.0.0, for a recent version of the Java programming language, is now available for download—check Major’s site so that you can download the tool and get started with mutation analysis for your JUnit test suites! If you are interested in Major, then you may also want to investigate the use of PIT, another mutation testing tool for programs written in the Java language.

Get the Gist!
Further Details

Interested in learning more about this topic? Since this blog post was written, my colleagues and students and I have published additional papers about mutation testing, with two noteworthy examples being (McCurdy, McMinn, and Kapfhammer 2016) and (McMinn et al. 2019) .

Return to Blog Post Listing

References

Just, René, Gregory M. Kapfhammer, and Franz Schweiggert. 2011a. “MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler.” In Proceedings of the 26th International Conference on Automated Software Engineering.
———. 2011b. “Using Conditional Mutation to Increase the Efficiency of Mutation Analysis.” In Proceedings of the 6th International Workshop on Automation of Software Test.
———. 2012. “Using Non-Redundant Mutation Operators and Test Suite Prioritization to Achieve Efficient and Scalable Mutation Analysis.” In Proceedings of the 23rd International Symposium on Software Reliability Engineering.
McCurdy, Colton J., Phil McMinn, and Gregory M. Kapfhammer. 2016. “Mrstudyr: Retrospectively Studying the Effectiveness of Mutant Reduction Techniques.” In Proceedings of the 32nd International Conference on Software Maintenance and Evolution.
McMinn, Phil, Chris J. Wright, Colton McCurdy, and Gregory M. Kapfhammer. 2019. “Automatic Detection and Removal of Ineffective Mutants for the Mutation Analysis of Relational Database Schemas.” Transactions on Software Engineering 45 (5).