# Choosing methods

This is a step-by-step for deciding which quality checks a codebase needs and in
what order to add them.

Different parts of a system need different amounts of checking. A payment path and
a marketing page carry nothing like the same risk, and a method that is cheap in
one place is expensive or pointless in another. Testing both to the same standard
either overspends on the page or underprotects the payments. So plan one part at a
time, not the whole codebase at once: a *part* is any piece with its own risks (a
payment path, a login flow, an admin screen, a data importer).

## Step 0: Name what is under test, and where its state lives

The artifact whose correctness is in question is not always the program. It may be
the data flowing through it — a pipeline, a batch job, an event stream — or a
configuration, a schema, or a trained model. A system written as application code
is not necessarily a system whose correctness lives in application code: a
transform can be fully tested and still forward a wrong payload, and every
service stays healthy while the product is wrong.

The state that artifact lives in is not always a repository either. Vendor and
SaaS configuration, infrastructure-as-code in a separate repository, a warehouse,
feature-flag rules, and CDN or edge rules each hold correctness the source tree
cannot show. A plan that covers the code while a consent gate lives in a vendor
console leaves that gate unverified.

Both answers scope every step in this procedure. The methods in this catalog are
classified by where their inputs come from and how they decide correctness, not by
substrate, so most of them apply to data, a configuration, or a schema as readily
as to code — once the artifact is named. A location no method reaches is a known
gap.

## Step 1: Choose the dimensions to focus on

For the part you are planning, decide what it most needs to get right, and put the
goals in order of importance. The five
[quality dimensions](https://quality.stereobooster.com/quality-dimensions.md):

- [functionality](https://quality.stereobooster.com/functionality.md): does it do the right thing?
- [security](https://quality.stereobooster.com/security.md): can it be attacked or abused?
- [reliability](https://quality.stereobooster.com/reliability.md): does it keep working, and recover, when something fails?
- [performance](https://quality.stereobooster.com/performance.md): is it fast enough under real load?
- [maintainability](https://quality.stereobooster.com/maintainability.md): can the next person change it without breaking it?

A payment path puts functionality and security on top; a nightly batch job leads
with reliability; an internal admin tool with maintainability. The order matters
because your effort follows it: the goals at the top earn the expensive methods
later, the ones at the bottom get only the cheap wins.

## Step 2: Set the guarantee bar, per dimension

For each of those goals, decide how much a passing check has to prove. This is not
one setting for the whole system: a ledger's money math may need a proof that it
can never go wrong, while the admin screen beside it just needs to work on
realistic inputs. The levels, weakest to strongest:

- **heuristic:** a rough check, better than nothing (a linter rule, a smoke test).
- **empirical:** tried on many real inputs, so you have good evidence but no proof.
- **sound:** **exhaustive** (every case in a bounded space) or **mathematical**
  (a proof), so that a pass means it *cannot* fail that way.

Only a **sound** bar is worth reaching for [formal methods](https://quality.stereobooster.com/formal.md)
and model checking; below it, they cost more than they return. Push a check deeper
as a component is used more widely, even with no regulator forcing it: the SQLite
core runs several independent test harnesses to 100% MC/DC (modified
condition/decision coverage) on deployment scale alone (SQLite Development Team 2025)[^sqlite2025]. Some
industries force the bar from outside: DO-178C's top level requires MC/DC and
structural coverage for flight software (RTCA 2011a)[^do178c], and medical-device work
carries its own risk-scaled rules (IEC 2006)[^iec62304].

## Step 3: Start from where the project already is

Whether the code is brand new or already running is a given, not a choice, and it
decides which tactic leads.

- **New or greenfield.** The cheapest wins come before any test: remove whole
  failure classes by construction. A memory-safe language retires the
  memory-corruption class; a typed schema at the boundary retires shape errors; a
  framework such as Rails or Django removes SQL injection and cross-site scripting
  through its ORM and auto-escaping, as long as the safe path is the default rather
  than an opt-in escape hatch (guard any escape hatch with a
  [static-analysis](https://quality.stereobooster.com/static-analysis.md) linter such as
  [Semgrep](https://semgrep.dev/)). An allocator-passing deterministic core makes
  [simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md) cheap
  later.
- **Existing project.** Start from what you already have: a suite, some tooling, a
  production history. Each boundary the work touches is pinned with
  **[approval (snapshot) tests](https://quality.stereobooster.com/snapshot-testing.md)** first and
  changed under them, with gradual typing or a parameterized-query layer added
  along the way.

## Step 4: Do the cheapest methods first

Turn on everything that costs about one flag, whatever dimension it serves:
[types](https://quality.stereobooster.com/types.md), linters, a sanitizer, the race detector,
dependency and [supply-chain](https://quality.stereobooster.com/supply-chain-hygiene.md) alerts,
and the [taint-analysis](https://quality.stereobooster.com/taint-analysis.md) rules your linter
already ships. These find bugs for almost no effort and remove work from every
step that follows, so take them even on a low-priority dimension.

## Step 5: Reuse the oracles you already have

The hard part of a test is knowing the right answer to check against. That
answer-source is called an **oracle**. When the code already gives you one, the
test almost writes itself, so look for these before writing checks by hand:

- **Round-trip:** `decode(encode(x)) == x`, free for any serializer.
- **Idempotence:** `f(f(x)) == f(x)`, free for anything meant to be safe to repeat.
- **Invariance:** a rule that must still hold after the operation, such as a
  preserved total or a still-valid schema.
- **A reference:** the slow version of a function, kept as the oracle its
  optimized replacement is diffed against.
- **A [decision table](https://quality.stereobooster.com/decision-tables.md)** for a handful of
  boolean inputs, among the cheapest formal methods and the one most teams overlook
  (Wayne 2018)[^wayne2018b].
- **A contract:** a precondition is already a property, checkable against generated
  inputs (Wayne 2017)[^wayne2017a].

Each of these holds for every input, not just the few cases a hand-written test
enumerates.

## Step 6: Locate a method for each remaining failure mode

For the failures where you have no oracle yet, match a method to the situation. The
[methods catalog](https://quality.stereobooster.com/methods.md) lists every method by the kind of input it
takes, the kind of oracle it uses, and the guarantee a pass earns; the
[axes](https://quality.stereobooster.com/axes.md) explain those keys. The common situations:

- **a reference implementation on hand** points to
  [differential testing](https://quality.stereobooster.com/differential-testing.md) (a
  reimplementation against the incumbent; a SIMD kernel against the scalar version
  it replaced);
- **several independent implementations, or a shared spec suite,** give an oracle
  from agreement (conformance suites, browser reftests, emulator test ROMs);
- **a mature domain with untrusted input** points to
  [fuzzing](https://quality.stereobooster.com/fuzzing.md), a free crash-oracle that needs no
  reference (a parser harness is a few dozen lines on [OSS-Fuzz](https://google.github.io/oss-fuzz/));
- **concurrency over a deterministic core** points to
  [simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md), and a
  protocol design to [TLA+](https://lamport.azurewebsites.net/tla/tla.html);
- **heavy state and I/O**, as in a typical web app, points to running the code
  against real dependencies (a real database, real services) instead of stand-ins,
  so a test covers the wiring and not just the logic;
- **a seam between parts of different owner or guarantee** points to a
  [contract](https://quality.stereobooster.com/contracts-and-runtime-assertions.md) or
  [schema and boundary validation](https://quality.stereobooster.com/schema-and-boundary-validation.md);
- **a large configuration or feature-flag space** points to
  [combinatorial (pairwise) testing](https://quality.stereobooster.com/combinatorial-testing.md);
- **a performance target** points to [profiling](https://quality.stereobooster.com/profiling.md),
  [microbenchmarking](https://quality.stereobooster.com/microbenchmarking.md), and
  [load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md);
- **output that is judged, not computed** (a ranking, a translation, generated
  media) points to the soft regime: eval-set statistics,
  [metamorphic relations](https://quality.stereobooster.com/metamorphic-testing.md), comparison
  against the prior version, and human judgment on samples. Here the guarantee caps
  at empirical.

## Step 7: Combine methods as the effect grows

No single method covers code that touches a lot, and the more a unit touches, the
more ways it can fail. The code's [effect](https://quality.stereobooster.com/effect.md) decides which
methods to combine:

| As the effect grows | Add these methods | What they buy |
| --- | --- | --- |
| **Pure logic** (no effect) | Examples, property tests, mutation testing | Property tests break the regression ceiling; mutation testing measures whether the examples detect anything. |
| **Local I/O and state** | + fault injection | Exercises the failure modes a stateful, side-effecting boundary opens up. |
| **Network and real dependencies** | + metamorphic or differential oracles, load and stress testing, chaos testing | Oracles reach the invariants enumeration cannot; load and stress testing find where throughput and latency break down; chaos testing surfaces the failures only real services produce. |
| **Production, unbounded nondeterminism** | + whole-system smoke tests, monitoring, canary or shadow traffic | A thin top layer that catches what slipped through, in the running system. |

The levels are not a strict ladder, since effect is a set of tags rather than a
sequence, but the progression is the common one.

## Step 8: Measure, then repeat until satisfied

Once a real suite exists, check that it works:
[mutation testing](https://quality.stereobooster.com/mutation-testing.md) tells you whether the
tests catch faults, and [coverage](https://quality.stereobooster.com/coverage.md) is the
gap-finder for a diff. Then loop back to step 6 for the failure modes still worth
covering. Two rules keep the loop honest:

- Always take the by-construction and near-free wins (steps 3 and 4), even on a
  minor dimension.
- Never leave a top-priority failure mode uncovered just because its method is
  expensive.

Stop when the next method would cost more than the failure it prevents. Base that
on this part's stakes, not on what similar projects happen to do. Write down what
you leave out as a known gap and stage it for when the part grows, its stakes rise,
or a bug class recurs.

## Scope

These steps cover the fit between methods and a part, not the choice of substrate.
The language, framework, and architecture behind step 3 also weigh familiarity, the
existing stack, and risk appetite; the steps inform that choice but do not make
it. A deterministic core makes simulation cheap. Where step 0 names the data
rather than the program, the dimensions and the methods are the same ones; what
changes is the vocabulary a data team will recognize them under
([ISO/IEC 25012](https://quality.stereobooster.com/quality-dimensions.md#data-quality)).

## Referenced by

- [Quality dimensions](https://quality.stereobooster.com/quality-dimensions.md) · Quality dimensions
- [Methods](https://quality.stereobooster.com/methods.md) · Methods
- [Software Quality](https://quality.stereobooster.com/index.md) · Overview

## References

[^sqlite2025]: SQLite Development Team. 2025. *[How SQLite Is Tested](https://www.sqlite.org/testing.html)*. <https://www.sqlite.org/testing.html>.
[^do178c]: RTCA. 2011a. *[DO-178C: Software Considerations in Airborne Systems and Equipment Certification](https://www.rtca.org/do-178/)*. <https://www.rtca.org/do-178/>.
[^iec62304]: IEC. 2006. *[IEC 62304: Medical Device Software — Software Life Cycle Processes](https://webstore.iec.ch/en/publication/6792)*. <https://webstore.iec.ch/en/publication/6792>.
[^wayne2018b]: Wayne, Hillel. 2018. *[Decision Tables](https://www.hillelwayne.com/post/decision-tables/)*. <https://www.hillelwayne.com/post/decision-tables/>.
[^wayne2017a]: Wayne, Hillel. 2017. *[Introduction to Contract Programming](https://www.hillelwayne.com/post/contracts/)*. <https://www.hillelwayne.com/post/contracts/>.

## Acronyms

- MC/DC — modified condition/decision coverage
- OSS — open-source software
- SIMD — single instruction, multiple data