Skip to content

Software Quality

Measuring test-suite effectiveness

Test-suite effectiveness is the field's term for how good the tests are: a suite's ability to detect faults (Inozemtseva and Holmes 2014; Zhang and Mesbah 2015)1 2.

Why a test catches a fault: the RIPR chain

A test reveals a fault only when four things happen in order. The model is RIPR (Ammann and Offutt 2017)3, which extends Voas's earlier PIE model (propagation, infection, execution) (Voas 1992)4:

  1. Reachability: the test executes the faulty location.
  2. Infection: executing it produces a wrong program state.
  3. Propagation: the wrong state reaches an observable output.
  4. Revealability: the suite's oracle (its assertions) checks that output and flags it.

Each factor is a probability, and they multiply down the chain, so a near-zero factor anywhere drives the whole detection probability to zero.

  • Coverage measures only reachability, the first factor. A covered line ran; nothing says it was infected, propagated, or checked. That is why coverage is a weak proxy for effectiveness, not a synonym for it.
  • The oracle is revealability, the last factor, and the one no coverage criterion can see.

The chain is conditioned on a present fault and on a run reaching it, so effectiveness is a measure of fault detection that exists only for methods that execute. A static method such as a type system or a proof produces no RIPR event and no effectiveness number.

The oracle problem

Revealability is hard to measure because, unlike reachability, it has no syntactic feature to count. The oracle is whatever decides pass or fail; Barr and colleagues survey the difficulty of getting one (Barr et al. 2015)5. Many systems have no cheap oracle for the property that matters at all.

Reachability: coverage and its criteria

Coverage measures reachability. Its criteria form an adequacy ladder: statement, branch, condition, up to multiple-condition; combinatorial testing measures t-way input coverage. All answer "did the suite get to enough of the space?" and all are cheap to compute.

But reachability is one factor of four, so a reach number predicts effectiveness only weakly. Holding suite size constant, the link between coverage and fault detection is weak; Inozemtseva and Holmes measured statement, decision, and modified-condition coverage against the mutation score, and found the more adequate criteria predicted no better than line coverage (Inozemtseva and Holmes 2014)1. Studies on real post-release bugs reach the same weak-to-moderate reading (Kochhar et al. 2017, 2015)6 7, while other work finds coverage adds some value beyond size, nonlinearly (Namin and Andrews 2009)8. Used as a target the number is worse than uninformative: a mandated percentage is exactly the claim the skeptical evidence undercuts.

Infection and propagation: mutation testing

Mutation testing exercises the whole chain. It changes the code in a small, deliberate way (a mutant) and checks whether some test fails; a mutant is killed only if a test reaches it, it infects the state, the infection propagates, and an assertion reveals it. The fraction killed is the mutation score, the closest single proxy for effectiveness. The equivalent-mutant problem caps it: some mutants are semantically identical to the original, so they can never be killed, and detecting them is undecidable in general. Redundant mutants inflate the score, which the subsuming- and minimal-mutant refinements correct.

Revealability on its own: oracle measures

Mutation is expensive: it re-runs the suite once per mutant. Cheaper measures read the revealability factor directly:

  • Checked coverage restricts coverage to statements whose result a dynamic backward slice shows reaching an assertion (Schuler and Zeller 2011)9. The gap between ordinary and checked coverage is the covered-but-unchecked code, quantified without mutating anything.
  • Assertion coverage and density count how many assertions a suite has and how much state they inspect; empirically the assertions, not the reach, are what track effectiveness (Zhang and Mesbah 2015)2.

Ground truth: real faults

Coverage, the mutation score, and the oracle measures are all proxies for one question: would the suite catch a real bug? That is measured directly on curated real-fault benchmarks, where a known historical bug is reverted and the suite is run against it. Defects4J is the standard Java corpus (Just, Jalali, and Ernst 2014)10; BugsInPy, Bugs.jar, and BugSwarm play the same role for other ecosystems. These also settle whether the cheaper proxies are valid: Andrews and colleagues found mutants behave enough like real faults to use in experiments (Andrews et al. 2005)11, and Just and colleagues found mutant detection correlated with real-fault detection (Just, Jalali, Inozemtseva, et al. 2014)12, while Papadakis and colleagues found that correlation weaker and confounded by suite size (Papadakis et al. 2018)13. The chain from coverage to mutants to real faults is real but loose at every link, so no single number ends the argument.

Other dimensions of test quality

Effectiveness is not the only sense in which tests are "good":

  • Diagnosability. Once a test fails, how precisely does the suite localize the cause? The DDU metric scores a suite on the density, diversity, and uniqueness of its coverage for spectrum-based fault localization (Perez et al. 2017)14.
  • Determinism. A flaky test gives an unstable verdict, which corrodes every effectiveness measure; reproducible execution is pursued by deterministic simulation testing.
  • Maintainability and cost. A suite can be hard to read or slow to run no matter how many bugs it catches. Cataloged test smells are the usual vocabulary for that, on evidence that runs both ways (Bavota et al. 2015; Panichella et al. 2022)15 16, and either way a smell is a maintainability claim, not a measure of effectiveness. How far the catalog's evidence reaches is set out under test smells.

The theory underneath

The question is old. Goodenough and Gerhart asked when a test set is adequate in principle (Goodenough and Gerhart 1975)17; Weyuker set out axioms any adequacy criterion ought to satisfy, and showed common ones violate several (Weyuker 1986)18; Zhu, Hall, and May survey the full space of coverage and adequacy criteria (Zhu et al. 1997)19. The recurring result is the one RIPR predicts: no purely structural criterion captures revealability.

What to measure when

  • For gap-finding (which code has no test at all), use coverage: it reads reachability, is sound in that direction, and costs nothing.
  • For effectiveness on code that matters, use mutation testing, or checked or assertion coverage where mutation is too slow.
  • For evaluating a tool or technique, measure against real-fault benchmarks; that is a research and calibration activity, not a per-commit one.

The error to avoid is reading a reachability number as an effectiveness number.

Referenced by

References

  1. Inozemtseva, Laura, and Reid Holmes. 2014. "Coverage Is Not Strongly Correlated with Test Suite Effectiveness." Proceedings of the 36th International Conference on Software Engineering (ICSE '14), 435–45. https://doi.org/10.1145/2568225.2568271. ↩↩

  2. Zhang, Yucheng, and Ali Mesbah. 2015. "Assertions Are Strongly Correlated with Test Suite Effectiveness." Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE '15), 214–24. https://doi.org/10.1145/2786805.2786858. ↩↩

  3. Ammann, Paul, and Jeff Offutt. 2017. Introduction to Software Testing. 2nd ed. Cambridge University Press. https://doi.org/10.1017/9781316771273. ↩

  4. Voas, Jeffrey M. 1992. "PIE: A Dynamic Failure-Based Technique." IEEE Transactions on Software Engineering 18 (8): 717–27. https://doi.org/10.1109/32.153381. ↩

  5. Barr, Earl T., Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. "The Oracle Problem in Software Testing: A Survey." IEEE Transactions on Software Engineering 41 (5): 507–25. https://doi.org/10.1109/TSE.2014.2372785. ↩

  6. Kochhar, Pavneet Singh, David Lo, Julia Lawall, and Nachiappan Nagappan. 2017. "Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects." IEEE Transactions on Reliability 66 (4): 1213–28. https://doi.org/10.1109/TR.2017.2727062. ↩

  7. Kochhar, Pavneet Singh, Ferdian Thung, and David Lo. 2015. "Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems." Proceedings of the 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER '15), 560–64. https://doi.org/10.1109/SANER.2015.7081877. ↩

  8. Namin, Akbar Siami, and James H. Andrews. 2009. "The Influence of Size and Coverage on Test Suite Effectiveness." Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA '09), 57–68. https://doi.org/10.1145/1572272.1572280. ↩

  9. Schuler, David, and Andreas Zeller. 2011. "Assessing Oracle Quality with Checked Coverage." Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation (ICST '11), 90–99. https://doi.org/10.1109/ICST.2011.32. ↩

  10. Just, René, Darioush Jalali, and Michael D. Ernst. 2014. "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs." Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA '14), 437–40. https://doi.org/10.1145/2610384.2628055. ↩

  11. Andrews, James H., Lionel C. Briand, and Yvan Labiche. 2005. "Is Mutation an Appropriate Tool for Testing Experiments?" Proceedings of the 27th International Conference on Software Engineering (ICSE '05), 402–11. https://doi.org/10.1109/ICSE.2005.1553583. ↩

  12. Just, René, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. "Are Mutants a Valid Substitute for Real Faults in Software Testing?" Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '14), 654–65. https://doi.org/10.1145/2635868.2635929. ↩

  13. Papadakis, Mike, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. "Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults." Proceedings of the 40th International Conference on Software Engineering (ICSE '18), 537–48. https://doi.org/10.1145/3180155.3180183. ↩

  14. Perez, Alexandre, Rui Abreu, and Arie van Deursen. 2017. "A Test-Suite Diagnosability Metric for Spectrum-Based Fault Localization Approaches." Proceedings of the 39th International Conference on Software Engineering (ICSE '17), 654–64. https://doi.org/10.1109/ICSE.2017.66. ↩

  15. Bavota, Gabriele, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and David W. Binkley. 2015. "Are Test Smells Really Harmful? An Empirical Study." Empirical Software Engineering 20 (4): 1052–94. https://doi.org/10.1007/s10664-014-9313-0. ↩

  16. Panichella, Annibale, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J. Hellendoorn. 2022. "Test Smells 20 Years Later: Detectability, Validity, and Reliability." Empirical Software Engineering 27 (7): 170. https://doi.org/10.1007/s10664-022-10207-5. ↩

  17. Goodenough, John B., and Susan L. Gerhart. 1975. "Toward a Theory of Test Data Selection." IEEE Transactions on Software Engineering SE-1 (2): 156–73. https://doi.org/10.1109/TSE.1975.6312836. ↩

  18. Weyuker, Elaine J. 1986. "Axiomatizing Software Test Data Adequacy." IEEE Transactions on Software Engineering 12 (12): 1128–38. https://doi.org/10.1109/TSE.1986.6313008. ↩

  19. Zhu, Hong, Patrick A. V. Hall, and John H. R. May. 1997. "Software Unit Test Coverage and Adequacy." ACM Computing Surveys 29 (4): 366–427. https://doi.org/10.1145/267580.267590. ↩