I recently hosted an episode of Software Engineering Radio called "Will McGugan on Text-Based User Interfaces"!

  • Home
  • Teaching
    • Overview
    • Data Abstraction
    • Operating Systems
  • Research
    • Overview
    • Papers
    • Presentations
  • Outreach
    • Software
    • Service
    • Blog
  • About
    • Biography
    • Schedule
    • Contact
    • Blog
    • Service
    • Papers
    • Presentations

Contents

  • Introduction
  • Understanding XMT
  • Research Insights
  • Future Directions

Using eXtreme mutation testing to discover hidden test suite weaknesses

post
mutation testing
developer productivity
Can tools detect hidden weaknesses in test suites?
Author

Gregory M. Kapfhammer

Published

2024

Introduction

Ever wondered if your test suite is truly good at finding bugs? What if there are hidden weaknesses that traditional testing methods miss? In the paper (Maton, Kapfhammer, and McMinn 2024a) , my colleagues and I explore how eXtreme mutation testing (XMT) can uncover these hidden weaknesses in your test suite. By identifying pseudo-tested methods — or those methods in your program that appear to be tested even though they are not — XMT can help you improve your test suite’s effectiveness and ensure more robust software. Let’s dive into how these XMT works and why it may be a game-changer for software testing!

Understanding XMT

An XMT tool automatically detects deficiencies in a test suite by deleting method bodies in covered code and observing whether the test suite detects their absence. If a method deletion goes unnoticed, then the method is labeled as “pseudo-tested,” meaning that the test suite calls the method but its result isn’t checked by the assertions, indicating a potential blind spot in the test suite. As an extension to XMT, our paper explores the use of the statement deletion mutation operator (SDL) to uncover pseudo-tested statements within methods.

Research Insights

Using the PseudoSweep tool (Maton, Kapfhammer, and McMinn 2024b) to perform mutation testing with the XMT and SDL operators on four Apache Commons projects and 23 projects randomly selected from the Maven Central Repository, this paper’s experiments find 722 different cases of pseudo-tested statements. Interestingly, the results show that 48% of pseudo-tested statements exist outside of pseudo-tested methods! This suggests that XMT alone is insufficient and should be combined with pseudo-tested statement evaluation to find testing oversights that a developer should tackle before using standard mutation testing.

Future Directions

While our research focused on Java programs, the concepts of XMT and pseudo-tested methods are broadly applicable. These techniques can be used in various programming languages to improve test suite effectiveness and ensure more robust software. We’re excited about the potential of combining XMT with other testing techniques to further enhance test suite robustness. Future work will involve extending our approach to different programming languages and exploring additional ways to detect and repair pseudo-tested methods.

Further Details

As we continue to explore how mutation testing can support the detection and enhanced assessment of pseudo-tested methods, your insights and suggestions are appreciated! If you have ideas or experiences related to assessing the quality of a test suite or finding testing blind spots, please contact me. Or, if you want to stay informed about new developments and blog posts related to mutation testing and other software testing topics, you can subscribe to my mailing list.

Return to Blog Post Listing

References

Maton, Megan, Gregory M. Kapfhammer, and Phil McMinn. 2024a. “Exploring Pseudo-Testedness: Empirically Evaluating Extreme Mutation Testing at the Statement Level.” In Proceedings of the 35th International Conference on Software Maintenance and Evolution.
———. 2024b. “PseudoSweep: A Pseudo-Tested Code Identifier.” In Proceedings of the 35th International Conference on Software Maintenance and Evolution – Tool Demonstrations Track.

GMK

Top