# Guarantee

The guarantee is how strong a claim a passing run makes: from a guess that
nothing obvious is broken, up to a proof that the failure cannot occur at all.
Unlike the other axes, its values are not distinct kinds but points on a
strength ordering.

Only one boundary on that ordering is sharp — **soundness**: whether a pass can
still miss the bug it targets. Everything finer is a matter of degree; within
each half the four tiers rank by strength of the *conclusion*, not by how much
code was exercised (*adequacy*):

<figure class="axis-tree" markdown>

- unsound
    - **heuristic**
    - **empirical**
- sound
    - **exhaustive**
    - **mathematical**

</figure>

- **Unsound.** A pass is evidence, not proof; the method can report "ok"
  while the bug is present.
    - **Heuristic.** The check is a fallible judgment: it can report "ok" with
      the fault in plain view. Smoke tests, exploratory testing. *Something
      obvious isn't broken right now.*
    - **Empirical.** The check is reliable but partial: the cases it ran
      genuinely passed, but they only sample the space. Most example tests,
      property tests, fuzzing, monitoring. *The cases that ran passed.*
      (Dijkstra: testing shows the presence of bugs, never their absence.)
- **Sound** *(within its scope)*. A pass proves the targeted failure cannot
  occur.
    - **Exhaustive (within a bound).** The check covers an explicitly bounded
      space in full, by enumeration. Combinatorial coverage of a small input
      space; bounded model checking. *Within these explicit bounds, no failure
      exists.*
    - **Mathematical.** The check covers the unbounded space by reasoning, not
      enumeration. Theorem proving; verified compilers and kernels. *Within the
      assumed semantics, no failure exists at all.*

Property-based testing sits on the edge: unsound at the level of any single run,
but the *property* is a claim about all inputs, a stronger conceptual statement
than an example test.

## Guarantee, effectiveness, and adequacy

These ideas about test quality are easy to run together.

- **Guarantee** is the strength of the claim a passing run makes: the per-method
  tier this axis assigns, from a guess (heuristic) to a proof (mathematical). The
  term is this project's; the underlying property is *soundness*.
- **Effectiveness** is a suite's ability to detect faults
  (Inozemtseva and Holmes 2014; Zhang and Mesbah 2015)[^inozemtseva2014] [^zhang2015], measured after the fact by mutation score or
  the
  [RIPR](https://quality.stereobooster.com/measuring-test-effectiveness.md) chain. It is the
  field's term for how good a set of tests is.
- **Adequacy** is a criterion-relative measure of test thoroughness (Zhu et al. 1997)[^zhu1997]: a
  suite is "X-adequate" if it meets criterion X, such as statement, branch,
  MC/DC (modified condition/decision coverage), or mutation. The criterion is
  both a stopping rule and a measurement.

They come apart pairwise:

- **Guarantee is not effectiveness.** A sound type system carries a sound
  guarantee for the errors it targets, yet has no effectiveness number at all
  against logic bugs: a static method never runs, so there are no faults-caught to
  count. A strong claim about a narrow target says nothing about faults outside it.
- **Adequacy is not effectiveness.** Meeting a coverage criterion does not predict
  fault detection: a high-coverage suite can still catch few bugs
  (Inozemtseva and Holmes 2014)[^inozemtseva2014], because coverage reads only the first RIPR link,
  reachability.
- **Adequacy refines guarantee, it does not set it.** The tier is the coarse
  warrant; adequacy is depth within a tier. Two methods in one tier can differ in
  adequacy: deterministic simulation testing and chaos engineering are both
  empirical, but the first explores its state space systematically while the
  second pokes at it by chance.

## Precision

Guarantee, effectiveness, and adequacy all measure one side of the ledger:
**recall**, the false-negative question of whether a pass can still hide a real
bug. The mirror property is **precision**, the false-positive question: when the
suite reports a failure, is a real bug behind it?

A test suite loses precision mainly through **flakiness**: a test that fails or
passes on the same code by chance reports a failure that usually has no bug
behind it. [Flaky tests](https://quality.stereobooster.com/flaky-tests.md) covers why such a
test is repaired rather than deleted. (Brittle, over-fitted assertions cost
precision the same way.)

Precision rarely appears beside the guarantee tiers. A check that cries wolf gets
ignored, so false alarms are hunted down first: flaky tests by quarantine, retry,
or repair; static-analysis
[false positives](https://quality.stereobooster.com/static-analysis.md) by high-signal rule
selection, where they otherwise kill adoption (Bessey et al. 2010; Johnson et al. 2013)[^bessey2010] [^johnson2013]. The
guarantee axis assumes a trustworthy verdict and asks the recall question that is
left.

## The RIPR parallel

Guarantee is the **warrant for effectiveness**. Effectiveness is the fault
detection a suite actually achieves; guarantee is how far a method's construction
lets a passing run vouch for it. The
[RIPR chain](https://quality.stereobooster.com/measuring-test-effectiveness.md) gives the shape:
of the four steps a test takes to catch a fault, a method controls only the two
ends. The **input** governs **reachability**, how much of the space a run reaches
and exercises; the **oracle** governs **revealability**, whether a verdict
surfaces a wrong result when it does. The chain's middle, whether a reached fault
corrupts the state and whether that corruption travels to where the oracle looks,
is a property of the specific bug rather than of the method, so no axis carries it
(Ammann and Offutt 2017; Voas 1992)[^ammann2017] [^voas1992].

So a method's guarantee is the warrant those two ends can carry, no stronger than
the weaker of them. This is a parallel, not an identity: RIPR scores one suite as
a probability, while the guarantee tier buckets a whole method, including methods
that never run.

## Not an independent axis

The other two axes, [input](https://quality.stereobooster.com/input.md) and [oracle](https://quality.stereobooster.com/oracle.md), are
independent: any input strategy combines with any oracle. Guarantee is not. The
strongest tier a method can reach is *capped* by both, at the weaker of the two
ceilings:

`guarantee ≤ min(ceiling(input), ceiling(oracle))`

- **Input caps how much of the space a pass speaks for.** Sampling (`live`
  production traffic, or random / coverage-guided `generative` input) can only
  ever be *empirical*: the runs that happened passed. A `fixed` input rises to
  *exhaustive* only when its finite domain is enumerated in full
  ([decision tables](https://quality.stereobooster.com/decision-tables.md)); a solver that
  reasons over all paths at once can reach *mathematical*.
- **Oracle caps how strong each verdict is.** A `judgment` oracle (human taste)
  tops out at *heuristic*; a `relation` oracle at *empirical*; only a total
  `equality` or `predicate` oracle can underwrite a proof. A `probabilistic`
  oracle splits by sub-kind: a sampled estimate caps at *empirical*, an exactly
  computed probability ([probabilistic model checking](https://quality.stereobooster.com/probabilistic-model-checking.md))
  at *exhaustive*.
- **The weaker ceiling binds.** A differential oracle could in principle prove
  equivalence, but run against `live` traffic it caps at *empirical*, a
  [parallel run](https://quality.stereobooster.com/parallel-run.md); the *same* oracle driven by
  a solver over all inputs reaches *mathematical*
  ([equivalence checking](https://quality.stereobooster.com/equivalence-checking.md)).

The bound is `≤`, not `=`: two methods with the same input and oracle can still
sit at different tiers. [Example tests](https://quality.stereobooster.com/example-tests.md) and
[decision tables](https://quality.stereobooster.com/decision-tables.md) both use fixed,
authored-equality pairs, yet one is empirical and the other exhaustive; the
slack is *coverage completeness*, whether the cases tried sample the space or
exhaust it. Because guarantee is derived from the other two rather than chosen
freely, the method catalog's combinatorial grid runs on input × oracle alone.

## Soundness, precisely

The sound/unsound line is the **soundness** property from program analysis: no
false negatives within the method's scope (see the soundness discussion on the
[types overview](https://quality.stereobooster.com/types.md)).

The fault-specific middle is why no sampling method crosses into *sound* by
adding cases. A run evaluates the two ends for the inputs it tried and leaves the
middle, whether each fault infects and propagates, unknown for every input it
skipped. Only coverage completeness settles it: enumerating a bounded domain in
full (the exhaustive tier) or reasoning over all inputs at once (the mathematical
tier).

**Assurance levels** are a *related but distinct* scale. DO-178C Design Assurance
Levels (A–E) (RTCA 2011a)[^do178c] and IEC 62304 software classes (A–C) (IEC 2006)[^iec62304] rank a
component by *failure severity* and prescribe *how much verification rigor* it
must meet, which dictates how far up the guarantee scale a component must climb.
The level tracks consequence, not strength: DO-178C reaches its top level via
testing plus MC/DC coverage, an *empirical* claim with high adequacy, and formal
proof became an allowed substitute for some objectives only with the DO-333
formal-methods supplement (RTCA 2011b)[^do333].

## Referenced by

- [Effect scope](https://quality.stereobooster.com/effect.md) · The axes
- [Oracle](https://quality.stereobooster.com/oracle.md) · The axes
- [The axes](https://quality.stereobooster.com/axes.md) · The axes
- [Combinatorial and pairwise testing](https://quality.stereobooster.com/combinatorial-testing.md) · Methods
- [Contracts as specifications](https://quality.stereobooster.com/contracts-as-specifications.md) · Methods
- [Methods](https://quality.stereobooster.com/methods.md) · Methods
- [Glossary](https://quality.stereobooster.com/glossary.md) · Overview

## References

[^inozemtseva2014]: Inozemtseva, Laura, and Reid Holmes. 2014. "[Coverage Is Not Strongly Correlated with Test Suite Effectiveness](https://www.cs.ubc.ca/~rtholmes/papers/icse_2014_inozemtseva.pdf)." *Proceedings of the 36th International Conference on Software Engineering (ICSE '14)*, 435–45. <https://doi.org/10.1145/2568225.2568271>.
[^zhang2015]: Zhang, Yucheng, and Ali Mesbah. 2015. "[Assertions Are Strongly Correlated with Test Suite Effectiveness](https://people.ece.ubc.ca/amesbah/resources/papers/fse15.pdf)." *Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE '15)*, 214–24. <https://doi.org/10.1145/2786805.2786858>.
[^zhu1997]: Zhu, Hong, Patrick A. V. Hall, and John H. R. May. 1997. "[Software Unit Test Coverage and Adequacy](https://www.cs.toronto.edu/~chechik/courses18/csc410/p366-zhu.pdf)." *ACM Computing Surveys* 29 (4): 366–427. <https://doi.org/10.1145/267580.267590>.
[^bessey2010]: Bessey, Al, Ken Block, Ben Chelf, et al. 2010. "[A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World](https://web.stanford.edu/~engler/BLOC-coverity.pdf)." *Communications of the ACM* 53 (2): 66–75. <https://doi.org/10.1145/1646353.1646374>.
[^johnson2013]: Johnson, Brittany, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. "[Why Don't Software Developers Use Static Analysis Tools to Find Bugs?](https://petertsehsun.github.io/soen7481/papers/icse13b.pdf)" *Proceedings of the 35th International Conference on Software Engineering (ICSE '13)* (San Francisco), 672–81. <https://doi.org/10.1109/ICSE.2013.6606613>.
[^ammann2017]: Ammann, Paul, and Jeff Offutt. 2017. *[Introduction to Software Testing](https://doi.org/10.1017/9781316771273)*. 2nd ed. Cambridge University Press. <https://doi.org/10.1017/9781316771273>.
[^voas1992]: Voas, Jeffrey M. 1992. "[PIE: A Dynamic Failure-Based Technique](https://cs.odu.edu/~mln/ltrs-pdfs/NASA-92-ieeeswe.jmv.pdf)." *IEEE Transactions on Software Engineering* 18 (8): 717–27. <https://doi.org/10.1109/32.153381>.
[^do178c]: RTCA. 2011a. *[DO-178C: Software Considerations in Airborne Systems and Equipment Certification](https://www.rtca.org/do-178/)*. <https://www.rtca.org/do-178/>.
[^iec62304]: IEC. 2006. *[IEC 62304: Medical Device Software — Software Life Cycle Processes](https://webstore.iec.ch/en/publication/6792)*. <https://webstore.iec.ch/en/publication/6792>.
[^do333]: RTCA. 2011b. *[DO-333: Formal Methods Supplement to DO-178C and DO-278A](https://www.rtca.org/do-178/)*. <https://www.rtca.org/do-178/>.

## Acronyms

- MC/DC — modified condition/decision coverage
- PIE — propagation, infection, execution
- RIPR — reachability, infection, propagation, revealability