Skip to content

Software Quality

Guarantee

The guarantee is how strong a claim a passing run makes: from a guess that nothing obvious is broken, up to a proof that the failure cannot occur at all. Unlike the other axes, its values are not distinct kinds but points on a strength ordering.

Only one boundary on that ordering is sharp — soundness: whether a pass can still miss the bug it targets. Everything finer is a matter of degree; within each half the four tiers rank by strength of the conclusion, not by how much code was exercised (adequacy):

  • unsound
    • heuristic
    • empirical
  • sound
    • exhaustive
    • mathematical
  • Unsound. A pass is evidence, not proof; the method can report "ok" while the bug is present.
    • Heuristic. The check is a fallible judgment: it can report "ok" with the fault in plain view. Smoke tests, exploratory testing. Something obvious isn't broken right now.
    • Empirical. The check is reliable but partial: the cases it ran genuinely passed, but they only sample the space. Most example tests, property tests, fuzzing, monitoring. The cases that ran passed. (Dijkstra: testing shows the presence of bugs, never their absence.)
  • Sound (within its scope). A pass proves the targeted failure cannot occur.
    • Exhaustive (within a bound). The check covers an explicitly bounded space in full, by enumeration. Combinatorial coverage of a small input space; bounded model checking. Within these explicit bounds, no failure exists.
    • Mathematical. The check covers the unbounded space by reasoning, not enumeration. Theorem proving; verified compilers and kernels. Within the assumed semantics, no failure exists at all.

Property-based testing sits on the edge: unsound at the level of any single run, but the property is a claim about all inputs, a stronger conceptual statement than an example test.

Guarantee, effectiveness, and adequacy

These ideas about test quality are easy to run together.

  • Guarantee is the strength of the claim a passing run makes: the per-method tier this axis assigns, from a guess (heuristic) to a proof (mathematical). The term is this project's; the underlying property is soundness.
  • Effectiveness is a suite's ability to detect faults (Inozemtseva and Holmes 2014; Zhang and Mesbah 2015)1 2, measured after the fact by mutation score or the RIPR chain. It is the field's term for how good a set of tests is.
  • Adequacy is a criterion-relative measure of test thoroughness (Zhu et al. 1997)3: a suite is "X-adequate" if it meets criterion X, such as statement, branch, MC/DC (modified condition/decision coverage), or mutation. The criterion is both a stopping rule and a measurement.

They come apart pairwise:

  • Guarantee is not effectiveness. A sound type system carries a sound guarantee for the errors it targets, yet has no effectiveness number at all against logic bugs: a static method never runs, so there are no faults-caught to count. A strong claim about a narrow target says nothing about faults outside it.
  • Adequacy is not effectiveness. Meeting a coverage criterion does not predict fault detection: a high-coverage suite can still catch few bugs (Inozemtseva and Holmes 2014)1, because coverage reads only the first RIPR link, reachability.
  • Adequacy refines guarantee, it does not set it. The tier is the coarse warrant; adequacy is depth within a tier. Two methods in one tier can differ in adequacy: deterministic simulation testing and chaos engineering are both empirical, but the first explores its state space systematically while the second pokes at it by chance.

Precision

Guarantee, effectiveness, and adequacy all measure one side of the ledger: recall, the false-negative question of whether a pass can still hide a real bug. The mirror property is precision, the false-positive question: when the suite reports a failure, is a real bug behind it?

A test suite loses precision mainly through flakiness: a test that fails or passes on the same code by chance reports a failure that usually has no bug behind it. Flaky tests covers why such a test is repaired rather than deleted. (Brittle, over-fitted assertions cost precision the same way.)

Precision rarely appears beside the guarantee tiers. A check that cries wolf gets ignored, so false alarms are hunted down first: flaky tests by quarantine, retry, or repair; static-analysis false positives by high-signal rule selection, where they otherwise kill adoption (Bessey et al. 2010; Johnson et al. 2013)4 5. The guarantee axis assumes a trustworthy verdict and asks the recall question that is left.

The RIPR parallel

Guarantee is the warrant for effectiveness. Effectiveness is the fault detection a suite actually achieves; guarantee is how far a method's construction lets a passing run vouch for it. The RIPR chain gives the shape: of the four steps a test takes to catch a fault, a method controls only the two ends. The input governs reachability, how much of the space a run reaches and exercises; the oracle governs revealability, whether a verdict surfaces a wrong result when it does. The chain's middle, whether a reached fault corrupts the state and whether that corruption travels to where the oracle looks, is a property of the specific bug rather than of the method, so no axis carries it (Ammann and Offutt 2017; Voas 1992)6 7.

So a method's guarantee is the warrant those two ends can carry, no stronger than the weaker of them. This is a parallel, not an identity: RIPR scores one suite as a probability, while the guarantee tier buckets a whole method, including methods that never run.

Not an independent axis

The other two axes, input and oracle, are independent: any input strategy combines with any oracle. Guarantee is not. The strongest tier a method can reach is capped by both, at the weaker of the two ceilings:

guarantee ≤ min(ceiling(input), ceiling(oracle))

  • Input caps how much of the space a pass speaks for. Sampling (live production traffic, or random / coverage-guided generative input) can only ever be empirical: the runs that happened passed. A fixed input rises to exhaustive only when its finite domain is enumerated in full (decision tables); a solver that reasons over all paths at once can reach mathematical.
  • Oracle caps how strong each verdict is. A judgment oracle (human taste) tops out at heuristic; a relation oracle at empirical; only a total equality or predicate oracle can underwrite a proof. A probabilistic oracle splits by sub-kind: a sampled estimate caps at empirical, an exactly computed probability (probabilistic model checking) at exhaustive.
  • The weaker ceiling binds. A differential oracle could in principle prove equivalence, but run against live traffic it caps at empirical, a parallel run; the same oracle driven by a solver over all inputs reaches mathematical (equivalence checking).

The bound is ≤, not =: two methods with the same input and oracle can still sit at different tiers. Example tests and decision tables both use fixed, authored-equality pairs, yet one is empirical and the other exhaustive; the slack is coverage completeness, whether the cases tried sample the space or exhaust it. Because guarantee is derived from the other two rather than chosen freely, the method catalog's combinatorial grid runs on input × oracle alone.

Soundness, precisely

The sound/unsound line is the soundness property from program analysis: no false negatives within the method's scope (see the soundness discussion on the types overview).

The fault-specific middle is why no sampling method crosses into sound by adding cases. A run evaluates the two ends for the inputs it tried and leaves the middle, whether each fault infects and propagates, unknown for every input it skipped. Only coverage completeness settles it: enumerating a bounded domain in full (the exhaustive tier) or reasoning over all inputs at once (the mathematical tier).

Assurance levels are a related but distinct scale. DO-178C Design Assurance Levels (A–E) (RTCA 2011a)8 and IEC 62304 software classes (A–C) (IEC 2006)9 rank a component by failure severity and prescribe how much verification rigor it must meet, which dictates how far up the guarantee scale a component must climb. The level tracks consequence, not strength: DO-178C reaches its top level via testing plus MC/DC coverage, an empirical claim with high adequacy, and formal proof became an allowed substitute for some objectives only with the DO-333 formal-methods supplement (RTCA 2011b)10.

Referenced by

References

  1. Inozemtseva, Laura, and Reid Holmes. 2014. "Coverage Is Not Strongly Correlated with Test Suite Effectiveness." Proceedings of the 36th International Conference on Software Engineering (ICSE '14), 435–45. https://doi.org/10.1145/2568225.2568271. ↩↩

  2. Zhang, Yucheng, and Ali Mesbah. 2015. "Assertions Are Strongly Correlated with Test Suite Effectiveness." Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE '15), 214–24. https://doi.org/10.1145/2786805.2786858. ↩

  3. Zhu, Hong, Patrick A. V. Hall, and John H. R. May. 1997. "Software Unit Test Coverage and Adequacy." ACM Computing Surveys 29 (4): 366–427. https://doi.org/10.1145/267580.267590. ↩

  4. Bessey, Al, Ken Block, Ben Chelf, et al. 2010. "A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World." Communications of the ACM 53 (2): 66–75. https://doi.org/10.1145/1646353.1646374. ↩

  5. Johnson, Brittany, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. "Why Don't Software Developers Use Static Analysis Tools to Find Bugs?" Proceedings of the 35th International Conference on Software Engineering (ICSE '13) (San Francisco), 672–81. https://doi.org/10.1109/ICSE.2013.6606613. ↩

  6. Ammann, Paul, and Jeff Offutt. 2017. Introduction to Software Testing. 2nd ed. Cambridge University Press. https://doi.org/10.1017/9781316771273. ↩

  7. Voas, Jeffrey M. 1992. "PIE: A Dynamic Failure-Based Technique." IEEE Transactions on Software Engineering 18 (8): 717–27. https://doi.org/10.1109/32.153381. ↩

  8. RTCA. 2011a. DO-178C: Software Considerations in Airborne Systems and Equipment Certification. https://www.rtca.org/do-178/. ↩

  9. IEC. 2006. IEC 62304: Medical Device Software — Software Life Cycle Processes. https://webstore.iec.ch/en/publication/6792. ↩

  10. RTCA. 2011b. DO-333: Formal Methods Supplement to DO-178C and DO-278A. https://www.rtca.org/do-178/. ↩