Skip to content

Software Quality

Measuring test-suite effectiveness

Test-suite effectiveness is the field's term for how good the tests are: a suite's ability to detect faults (Inozemtseva and Holmes 2014; Zhang and Mesbah 2015)1 2.

Why a test catches a fault: the RIPR chain

A test reveals a fault only when four things happen in order. The model is RIPR, the fault/failure model Morell and Offutt developed independently, extended with revealability (Ammann and Offutt 2017)3; Voas's PIE analysis estimates its first three steps (execution, infection, propagation) as probabilities (Voas 1992)4:

  1. Reachability: the test executes the faulty location.
  2. Infection: executing it produces a wrong program state.
  3. Propagation: the wrong state reaches an observable output.
  4. Revealability: the suite's oracle (its assertions) checks that output and flags it.

Each factor is the probability of its step given the step before, so they multiply down the chain, and a near-zero factor anywhere drives the whole detection probability to zero.

  • Coverage measures only reachability, the first factor. A covered line ran; nothing says it was infected, propagated, or checked. That is why coverage is a weak proxy for effectiveness, not a synonym for it.
  • The oracle is revealability, the last factor, and the one no coverage criterion can see.

The chain is conditioned on a present fault and on a run reaching it, so effectiveness is a measure of fault detection that exists only for methods that execute. A static method such as a type system or a proof produces no RIPR event and no effectiveness number.

The oracle problem

Revealability is hard to measure because, unlike reachability, it has no syntactic feature to count. The oracle is whatever decides pass or fail; Barr and colleagues survey the difficulty of getting one (Barr et al. 2015)5. Many systems have no cheap oracle for the property that matters at all.

Reachability: coverage and its criteria

Coverage criteria form an adequacy ladder: statement, branch, condition, up to multiple-condition; combinatorial testing measures t-way input coverage. All answer "did the suite get to enough of the space?" and all are cheap to compute.

Holding suite size constant, the correlation between coverage and fault detection is low to moderate; Inozemtseva and Holmes measured statement, decision, and modified-condition coverage against the mutation score, and found the type of coverage made little difference to the strength of the relationship (Inozemtseva and Holmes 2014)1. On real bugs the results split: coverage correlated moderately to strongly with killing real bugs in two large Java systems (Kochhar et al. 2015)6, but had an insignificant correlation with post-release bugs across open-source projects (Kochhar et al. 2017)7. Other work finds coverage adds some value beyond size, nonlinearly (Namin and Andrews 2009)8. Inozemtseva and Holmes conclude that a fixed coverage value used as a quality target is unlikely to produce an effective suite (Inozemtseva and Holmes 2014)1.

Infection and propagation: mutation testing

Mutation testing exercises the whole chain. It changes the code in a small, deliberate way (a mutant) and checks whether some test fails; a mutant is killed only if a test reaches it, it infects the state, the infection propagates, and an assertion reveals it. The fraction killed is the mutation score. The equivalent-mutant problem caps it: some mutants are semantically identical to the original, so they can never be killed, and detecting them is undecidable in general. Redundant mutants inflate the score, which the subsuming- and minimal-mutant refinements correct.

Revealability on its own: oracle measures

Mutation is expensive: it re-runs the suite once per mutant. Cheaper measures read the revealability factor directly:

  • Checked coverage restricts coverage to statements whose result a dynamic backward slice shows reaching an assertion (Schuler and Zeller 2011)9. The gap between ordinary and checked coverage is the covered-but-unchecked code, quantified without mutating anything.
  • Assertion quantity and assertion coverage count how many assertions a suite has and how much of the code they check; empirically the assertions, not reachability, are what track effectiveness (Zhang and Mesbah 2015)2.

Ground truth: real faults

Coverage, the mutation score, and the oracle measures are all proxies for one question: would the suite catch a real bug? That is measured directly on curated real-fault benchmarks, where a known historical bug is reverted and the suite is run against it. Defects4J is the standard Java corpus (Just, Jalali, and Ernst 2014)10; Bugs.jar is a second Java corpus, BugsInPy covers Python, and BugSwarm mines Java and Python failures from CI builds. These also settle whether the cheaper proxies are valid: Andrews and colleagues found mutants behave enough like real faults to use in experiments (Andrews et al. 2005)11, and Just and colleagues found mutant detection correlated with real-fault detection (Just, Jalali, Inozemtseva, et al. 2014)12, while Papadakis and colleagues found that correlation weaker and confounded by suite size (Papadakis et al. 2018)13. The chain from coverage to mutants to real faults is real but loose at every link, so no single number ends the argument.

Other dimensions of test quality

Effectiveness is not the only sense in which tests are "good":

  • Diagnosability. Once a test fails, how precisely does the suite localize the cause? The DDU metric scores a suite on the density, diversity, and uniqueness of its coverage for spectrum-based fault localization (Perez et al. 2017)14.
  • Determinism. A flaky test gives an unstable verdict, which corrodes every effectiveness measure; reproducible execution is pursued by deterministic simulation testing.
  • Maintainability and cost. A suite can be hard to read or slow to run no matter how many bugs it catches. Cataloged test smells are the usual vocabulary for that, and a smell is a maintainability claim, not a measure of effectiveness.

The theory underneath

The question is old. Goodenough and Gerhart asked when a test set is adequate in principle (Goodenough and Gerhart 1975)15; Weyuker set out axioms any adequacy criterion ought to satisfy, and showed common ones violate several (Weyuker 1986)16; Zhu, Hall, and May survey the full space of coverage and adequacy criteria (Zhu et al. 1997)17. The recurring result is the one RIPR predicts: no purely structural criterion captures revealability.

What to measure when

  • For gap-finding (which code has no test at all), use coverage: it reads reachability and is sound in that direction.
  • For effectiveness on code that matters, use mutation testing, or checked or assertion coverage where mutation is too slow.
  • For evaluating a tool or technique, measure against real-fault benchmarks; that is a research and calibration activity, not a per-commit one.

Referenced by

References

  1. Inozemtseva, Laura, and Reid Holmes. 2014. "Coverage Is Not Strongly Correlated with Test Suite Effectiveness." Proceedings of the 36th International Conference on Software Engineering (ICSE '14), 435–45. https://doi.org/10.1145/2568225.2568271. ↩↩↩

  2. Zhang, Yucheng, and Ali Mesbah. 2015. "Assertions Are Strongly Correlated with Test Suite Effectiveness." Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE '15), 214–24. https://doi.org/10.1145/2786805.2786858. ↩↩

  3. Ammann, Paul, and Jeff Offutt. 2017. Introduction to Software Testing. 2nd ed. Cambridge University Press. https://doi.org/10.1017/9781316771273. ↩

  4. Voas, Jeffrey M. 1992. "PIE: A Dynamic Failure-Based Technique." IEEE Transactions on Software Engineering 18 (8): 717–27. https://doi.org/10.1109/32.153381. ↩

  5. Barr, Earl T., Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. "The Oracle Problem in Software Testing: A Survey." IEEE Transactions on Software Engineering 41 (5): 507–25. https://doi.org/10.1109/TSE.2014.2372785. ↩

  6. Kochhar, Pavneet Singh, Ferdian Thung, and David Lo. 2015. "Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems." Proceedings of the 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER '15), 560–64. https://doi.org/10.1109/SANER.2015.7081877. ↩

  7. Kochhar, Pavneet Singh, David Lo, Julia Lawall, and Nachiappan Nagappan. 2017. "Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects." IEEE Transactions on Reliability 66 (4): 1213–28. https://doi.org/10.1109/TR.2017.2727062. ↩

  8. Namin, Akbar Siami, and James H. Andrews. 2009. "The Influence of Size and Coverage on Test Suite Effectiveness." Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA '09), 57–68. https://doi.org/10.1145/1572272.1572280. ↩

  9. Schuler, David, and Andreas Zeller. 2011. "Assessing Oracle Quality with Checked Coverage." Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation (ICST '11), 90–99. https://doi.org/10.1109/ICST.2011.32. ↩

  10. Just, René, Darioush Jalali, and Michael D. Ernst. 2014. "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs." Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA '14), 437–40. https://doi.org/10.1145/2610384.2628055. ↩

  11. Andrews, James H., Lionel C. Briand, and Yvan Labiche. 2005. "Is Mutation an Appropriate Tool for Testing Experiments?" Proceedings of the 27th International Conference on Software Engineering (ICSE '05), 402–11. https://doi.org/10.1109/ICSE.2005.1553583. ↩

  12. Just, René, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. "Are Mutants a Valid Substitute for Real Faults in Software Testing?" Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '14), 654–65. https://doi.org/10.1145/2635868.2635929. ↩

  13. Papadakis, Mike, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. "Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults." Proceedings of the 40th International Conference on Software Engineering (ICSE '18), 537–48. https://doi.org/10.1145/3180155.3180183. ↩

  14. Perez, Alexandre, Rui Abreu, and Arie van Deursen. 2017. "A Test-Suite Diagnosability Metric for Spectrum-Based Fault Localization Approaches." Proceedings of the 39th International Conference on Software Engineering (ICSE '17), 654–64. https://doi.org/10.1109/ICSE.2017.66. ↩

  15. Goodenough, John B., and Susan L. Gerhart. 1975. "Toward a Theory of Test Data Selection." IEEE Transactions on Software Engineering SE-1 (2): 156–73. https://doi.org/10.1109/TSE.1975.6312836. ↩

  16. Weyuker, Elaine J. 1986. "Axiomatizing Software Test Data Adequacy." IEEE Transactions on Software Engineering 12 (12): 1128–38. https://doi.org/10.1109/TSE.1986.6313008. ↩

  17. Zhu, Hong, Patrick A. V. Hall, and John H. R. May. 1997. "Software Unit Test Coverage and Adequacy." ACM Computing Surveys 29 (4): 366–427. https://doi.org/10.1145/267580.267590. ↩