# Measuring test-suite effectiveness

Test-suite **effectiveness** is the field's term for how good the tests are: a
suite's ability to detect faults (Inozemtseva and Holmes 2014; Zhang and Mesbah 2015)[^inozemtseva2014] [^zhang2015].

## Why a test catches a fault: the RIPR chain

A test reveals a fault only when four things happen in order. The model is
**RIPR** (Ammann and Offutt 2017)[^ammann2017], which extends Voas's earlier PIE model (propagation,
infection, execution) (Voas 1992)[^voas1992]:

1. **Reachability**: the test executes the faulty location.
2. **Infection**: executing it produces a wrong program state.
3. **Propagation**: the wrong state reaches an observable output.
4. **Revealability**: the suite's *oracle* (its assertions) checks that output
   and flags it.

Each factor is a probability, and they *multiply* down the chain, so a near-zero
factor anywhere drives the whole detection probability to zero.

- **Coverage measures only reachability**, the first factor. A covered line
  *ran*; nothing says it was infected, propagated, or checked. That is why
  coverage is a weak proxy for effectiveness, not a synonym for it.
- **The oracle is revealability**, the last factor, and the one no coverage
  criterion can see.

The chain is conditioned on a present fault and on a run reaching it, so
effectiveness is a measure of fault detection that exists only for methods that
execute. A static method such as a type system or a proof produces no RIPR event
and no effectiveness number.

## The oracle problem

Revealability is hard to measure because, unlike reachability, it has no
syntactic feature to count. The oracle is whatever decides pass or fail; Barr
and colleagues survey the difficulty of getting one (Barr et al. 2015)[^barr2015]. Many systems
have no cheap oracle for the property that matters at all.

## Reachability: coverage and its criteria

[Coverage](https://quality.stereobooster.com/coverage.md) measures reachability. Its criteria form an
**adequacy** ladder: statement, branch, condition, up to multiple-condition;
[combinatorial testing](https://quality.stereobooster.com/combinatorial-testing.md) measures *t*-way input
coverage. All answer "did the suite get to enough of the space?" and all are
cheap to compute.

But reachability is one factor of four, so a reach number predicts effectiveness
only weakly. Holding suite size constant, the link between coverage and fault
detection is weak; Inozemtseva and Holmes measured statement, decision, and
modified-condition coverage against the mutation score, and found the more
adequate criteria predicted no better than line coverage (Inozemtseva and Holmes 2014)[^inozemtseva2014].
Studies on real post-release bugs reach the same weak-to-moderate reading
(Kochhar et al. 2017, 2015)[^kochhar2017] [^kochhar2015], while other work finds coverage adds some value
beyond size, nonlinearly (Namin and Andrews 2009)[^namin2009]. Used as a *target* the number is worse
than uninformative: a mandated percentage is exactly the claim the skeptical
evidence undercuts.

## Infection and propagation: mutation testing

[Mutation testing](https://quality.stereobooster.com/mutation-testing.md) exercises the *whole* chain. It changes
the code in a small, deliberate way (a mutant) and checks whether some test
fails; a mutant is killed only if a test reaches it, it infects the state, the
infection propagates, and an assertion reveals it. The fraction killed is the
**mutation score**, the closest single proxy for effectiveness. The
*equivalent-mutant problem* caps it: some mutants are semantically identical to
the original, so they can never be killed, and detecting them is undecidable in
general. Redundant mutants inflate the score, which the subsuming- and
minimal-mutant refinements correct.

## Revealability on its own: oracle measures

Mutation is expensive: it re-runs the suite once per mutant. Cheaper measures
read the revealability factor directly:

- **Checked coverage** restricts coverage to statements whose result a dynamic
  backward slice shows reaching an assertion (Schuler and Zeller 2011)[^schuler2011]. The gap between
  ordinary and checked coverage *is* the covered-but-unchecked code, quantified
  without mutating anything.
- **Assertion coverage and density** count how many assertions a suite has and
  how much state they inspect; empirically the assertions, not the reach, are
  what track effectiveness (Zhang and Mesbah 2015)[^zhang2015].

## Ground truth: real faults

Coverage, the mutation score, and the oracle measures are all proxies for one
question: would the suite catch a *real* bug? That is measured directly on
curated real-fault benchmarks, where a known historical bug is reverted and the
suite is run against it. Defects4J is the standard Java corpus (Just, Jalali, and Ernst 2014)[^just2014a];
BugsInPy, Bugs.jar, and BugSwarm play the same role for other ecosystems. These
also settle whether the cheaper proxies are valid: Andrews and colleagues found
mutants behave enough like real faults to use in experiments (Andrews et al. 2005)[^andrews2005], and
Just and colleagues found mutant detection correlated with real-fault detection
(Just, Jalali, Inozemtseva, et al. 2014)[^just2014], while Papadakis and colleagues found that correlation weaker and
confounded by suite size (Papadakis et al. 2018)[^papadakis2018]. The chain from coverage to mutants to
real faults is real but loose at every link, so no single number ends the
argument.

## Other dimensions of test quality

Effectiveness is not the only sense in which tests are "good":

- **Diagnosability.** Once a test fails, how precisely does the suite localize
  the cause? The DDU metric scores a suite on the density, diversity, and
  uniqueness of its coverage for spectrum-based fault localization (Perez et al. 2017)[^perez2017].
- **Determinism.** A [flaky test](https://quality.stereobooster.com/flaky-tests.md) gives an unstable verdict,
  which corrodes every effectiveness measure; reproducible execution is pursued
  by [deterministic simulation
  testing](https://quality.stereobooster.com/deterministic-simulation-testing.md).
- **Maintainability and cost.** A suite can be hard to read or slow to run no
  matter how many bugs it catches. Cataloged *test smells* are the usual
  vocabulary for that, on evidence that runs both ways
  (Bavota et al. 2015; Panichella et al. 2022)[^bavota2015] [^panichella2022], and either way a smell is a maintainability
  claim, not a measure of effectiveness. How far the catalog's evidence reaches
  is set out under [test smells](https://quality.stereobooster.com/test-smells.md).

## The theory underneath

The question is old. Goodenough and Gerhart asked when a test set is adequate in
principle (Goodenough and Gerhart 1975)[^goodenough1975]; Weyuker set out axioms any adequacy criterion ought
to satisfy, and showed common ones violate several (Weyuker 1986)[^weyuker1986]; Zhu, Hall, and
May survey the full space of coverage and adequacy criteria (Zhu et al. 1997)[^zhu1997]. The
recurring result is the one RIPR predicts: no purely structural criterion
captures revealability.

## What to measure when

- For **gap-finding** (which code has no test at all), use coverage: it reads
  reachability, is sound in that direction, and costs nothing.
- For **effectiveness** on code that matters, use [mutation
  testing](https://quality.stereobooster.com/mutation-testing.md), or checked or assertion coverage where mutation
  is too slow.
- For **evaluating a tool or technique**, measure against real-fault benchmarks;
  that is a research and calibration activity, not a per-commit one.

The error to avoid is reading a reachability number as an effectiveness number.

## Referenced by

- [Guarantee](https://quality.stereobooster.com/guarantee.md) · The axes
- [Coverage](https://quality.stereobooster.com/coverage.md) · Methods
- [Refactoring practice](https://quality.stereobooster.com/refactoring-practice.md) · Methods
- [Regression-suite management](https://quality.stereobooster.com/regression-management.md) · Methods
- [Test suite minimization](https://quality.stereobooster.com/test-suite-minimization.md) · Methods
- [Testing machine-learning systems](https://quality.stereobooster.com/testing-ml-systems.md) · Methods
- [The test suite as an object](https://quality.stereobooster.com/test-suite.md) · Methods
- [Test smells](https://quality.stereobooster.com/test-smells.md) · Conventional
- [Glossary](https://quality.stereobooster.com/glossary.md) · Overview

## References

[^inozemtseva2014]: Inozemtseva, Laura, and Reid Holmes. 2014. "[Coverage Is Not Strongly Correlated with Test Suite Effectiveness](https://www.cs.ubc.ca/~rtholmes/papers/icse_2014_inozemtseva.pdf)." *Proceedings of the 36th International Conference on Software Engineering (ICSE '14)*, 435–45. <https://doi.org/10.1145/2568225.2568271>.
[^zhang2015]: Zhang, Yucheng, and Ali Mesbah. 2015. "[Assertions Are Strongly Correlated with Test Suite Effectiveness](https://people.ece.ubc.ca/amesbah/resources/papers/fse15.pdf)." *Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE '15)*, 214–24. <https://doi.org/10.1145/2786805.2786858>.
[^ammann2017]: Ammann, Paul, and Jeff Offutt. 2017. *[Introduction to Software Testing](https://doi.org/10.1017/9781316771273)*. 2nd ed. Cambridge University Press. <https://doi.org/10.1017/9781316771273>.
[^voas1992]: Voas, Jeffrey M. 1992. "[PIE: A Dynamic Failure-Based Technique](https://cs.odu.edu/~mln/ltrs-pdfs/NASA-92-ieeeswe.jmv.pdf)." *IEEE Transactions on Software Engineering* 18 (8): 717–27. <https://doi.org/10.1109/32.153381>.
[^barr2015]: Barr, Earl T., Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. "[The Oracle Problem in Software Testing: A Survey](https://ieeexplore.ieee.org/ielx7/32/7106034/06963470.pdf)." *IEEE Transactions on Software Engineering* 41 (5): 507–25. <https://doi.org/10.1109/TSE.2014.2372785>.
[^kochhar2017]: Kochhar, Pavneet Singh, David Lo, Julia Lawall, and Nachiappan Nagappan. 2017. "[Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects](https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=4840&context=sis_research)." *IEEE Transactions on Reliability* 66 (4): 1213–28. <https://doi.org/10.1109/TR.2017.2727062>.
[^kochhar2015]: Kochhar, Pavneet Singh, Ferdian Thung, and David Lo. 2015. "[Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems](http://www.mysmu.edu/faculty/davidlo/papers/saner15-coverage.pdf)." *Proceedings of the 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER '15)*, 560–64. <https://doi.org/10.1109/SANER.2015.7081877>.
[^namin2009]: Namin, Akbar Siami, and James H. Andrews. 2009. "[The Influence of Size and Coverage on Test Suite Effectiveness](https://www.researchgate.net/profile/Akbar-Siami-Namin/publication/220854552_The_influence_of_size_and_coverage_on_test_suite_effectiveness/links/577164e508ae0b3a3b7d6e5d/The-influence-of-size-and-coverage-on-test-suite-effectiveness.pdf)." *Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA '09)*, 57–68. <https://doi.org/10.1145/1572272.1572280>.
[^schuler2011]: Schuler, David, and Andreas Zeller. 2011. "[Assessing Oracle Quality with Checked Coverage](https://www.st.cs.uni-saarland.de/publications/files/schuler-icst-2011.pdf)." *Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation (ICST '11)*, 90–99. <https://doi.org/10.1109/ICST.2011.32>.
[^just2014a]: Just, René, Darioush Jalali, and Michael D. Ernst. 2014. "[Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs](https://dl.acm.org/doi/pdf/10.1145/2610384.2628055)." *Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA '14)*, 437–40. <https://doi.org/10.1145/2610384.2628055>.
[^andrews2005]: Andrews, James H., Lionel C. Briand, and Yvan Labiche. 2005. "[Is Mutation an Appropriate Tool for Testing Experiments?](https://bowringj.people.charleston.edu/classes/csci%20362/docs/MutationTesting.pdf)" *Proceedings of the 27th International Conference on Software Engineering (ICSE '05)*, 402–11. <https://doi.org/10.1109/ICSE.2005.1553583>.
[^just2014]: Just, René, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. "[Are Mutants a Valid Substitute for Real Faults in Software Testing?](https://homes.cs.washington.edu/~rjust/publ/mutants_real_faults_tr_2014.pdf)" *Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '14)*, 654–65. <https://doi.org/10.1145/2635868.2635929>.
[^papadakis2018]: Papadakis, Mike, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. "[Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults](https://core.ac.uk/download/154760839.pdf)." *Proceedings of the 40th International Conference on Software Engineering (ICSE '18)*, 537–48. <https://doi.org/10.1145/3180155.3180183>.
[^perez2017]: Perez, Alexandre, Rui Abreu, and Arie van Deursen. 2017. "[A Test-Suite Diagnosability Metric for Spectrum-Based Fault Localization Approaches](https://pure.tudelft.nl/ws/files/11926562/TUD_SERG_2017_004.pdf)." *Proceedings of the 39th International Conference on Software Engineering (ICSE '17)*, 654–64. <https://doi.org/10.1109/ICSE.2017.66>.
[^bavota2015]: Bavota, Gabriele, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and David W. Binkley. 2015. "[Are Test Smells Really Harmful? An Empirical Study](https://link.springer.com/content/pdf/10.1007/s10664-014-9313-0.pdf)." *Empirical Software Engineering* 20 (4): 1052–94. <https://doi.org/10.1007/s10664-014-9313-0>.
[^panichella2022]: Panichella, Annibale, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J. Hellendoorn. 2022. "[Test Smells 20 Years Later: Detectability, Validity, and Reliability](https://pure.tudelft.nl/ws/portalfiles/portal/137994226/s10664_022_10207_5.pdf)." *Empirical Software Engineering* 27 (7): 170. <https://doi.org/10.1007/s10664-022-10207-5>.
[^goodenough1975]: Goodenough, John B., and Susan L. Gerhart. 1975. "[Toward a Theory of Test Data Selection](https://dl.acm.org/doi/pdf/10.1145/390016.808473)." *IEEE Transactions on Software Engineering* SE-1 (2): 156–73. <https://doi.org/10.1109/TSE.1975.6312836>.
[^weyuker1986]: Weyuker, Elaine J. 1986. "[Axiomatizing Software Test Data Adequacy](https://archive.org/download/axiomatizingsoft00weyu/axiomatizingsoft00weyu.pdf)." *IEEE Transactions on Software Engineering* 12 (12): 1128–38. <https://doi.org/10.1109/TSE.1986.6313008>.
[^zhu1997]: Zhu, Hong, Patrick A. V. Hall, and John H. R. May. 1997. "[Software Unit Test Coverage and Adequacy](https://www.cs.toronto.edu/~chechik/courses18/csc410/p366-zhu.pdf)." *ACM Computing Surveys* 29 (4): 366–427. <https://doi.org/10.1145/267580.267590>.

## Acronyms

- DDU — density, diversity, uniqueness
- PIE — propagation, infection, execution
- RIPR — reachability, infection, propagation, revealability