Skip to content

Software Quality

Choosing methods

A step-by-step for deciding which quality checks to add, and in what order.

This is a step-by-step for deciding which quality checks a codebase needs and in what order to add them.

Different parts of a system need different amounts of checking. A payment path and a marketing page carry nothing like the same risk, and a method that is cheap in one place is expensive or pointless in another. Testing both to the same standard either overspends on the page or underprotects the payments. So plan one part at a time, not the whole codebase at once: a part is any piece with its own risks (a payment path, a login flow, an admin screen, a data importer).

Step 0: Name what is under test, and where its state lives

The artifact whose correctness is in question is not always the program. It may be the data flowing through it — a pipeline, a batch job, an event stream — or a configuration, a schema, or a trained model. A system written as application code is not necessarily a system whose correctness lives in application code: a transform can be fully tested and still forward a wrong payload, and every service stays healthy while the product is wrong.

The state that artifact lives in is not always a repository either. Vendor and SaaS configuration, infrastructure-as-code in a separate repository, a warehouse, feature-flag rules, and CDN or edge rules each hold correctness the source tree cannot show. A plan that covers the code while a consent gate lives in a vendor console leaves that gate unverified.

Both answers scope every step in this procedure. The methods in this catalog are classified by where their inputs come from and how they decide correctness, not by substrate, so most of them apply to data, a configuration, or a schema as readily as to code — once the artifact is named. A location no method reaches is a known gap.

Step 1: Choose the dimensions to focus on

For the part you are planning, decide what it most needs to get right, and put the goals in order of importance. The five quality dimensions:

A payment path puts functionality and security on top; a nightly batch job leads with reliability; an internal admin tool with maintainability. The order matters because your effort follows it: the goals at the top earn the expensive methods later, the ones at the bottom get only the cheap wins.

Step 2: Set the guarantee bar, per dimension

For each of those goals, decide how much a passing check has to prove. This is not one setting for the whole system: a ledger's money math may need a proof that it can never go wrong, while the admin screen beside it just needs to work on realistic inputs. The levels, weakest to strongest:

  • heuristic: a rough check, better than nothing (a linter rule, a smoke test).
  • empirical: tried on many real inputs, so you have good evidence but no proof.
  • sound: exhaustive (every case in a bounded space) or mathematical (a proof), so that a pass means it cannot fail that way.

Only a sound bar is worth reaching for formal methods and model checking; below it, they cost more than they return. Push a check deeper as a component is used more widely, even with no regulator forcing it: the SQLite core runs several independent test harnesses to 100% MC/DC (modified condition/decision coverage) on deployment scale alone (SQLite Development Team 2025)1. Some industries force the bar from outside: DO-178C's top level requires MC/DC and structural coverage for flight software (RTCA 2011a)2, and medical-device work carries its own risk-scaled rules (IEC 2006)3.

Step 3: Start from where the project already is

Whether the code is brand new or already running is a given, not a choice, and it decides which tactic leads.

  • New or greenfield. The cheapest wins come before any test: remove whole failure classes by construction. A memory-safe language retires the memory-corruption class; a typed schema at the boundary retires shape errors; a framework such as Rails or Django removes SQL injection and cross-site scripting through its ORM and auto-escaping, as long as the safe path is the default rather than an opt-in escape hatch (guard any escape hatch with a static-analysis linter such as Semgrep). An allocator-passing deterministic core makes simulation testing cheap later.
  • Existing project. Start from what you already have: a suite, some tooling, a production history. Each boundary the work touches is pinned with approval (snapshot) tests first and changed under them, with gradual typing or a parameterized-query layer added along the way.

Step 4: Do the cheapest methods first

Turn on everything that costs about one flag, whatever dimension it serves: types, linters, a sanitizer, the race detector, dependency and supply-chain alerts, and the taint-analysis rules your linter already ships. These find bugs for almost no effort and remove work from every step that follows, so take them even on a low-priority dimension.

Step 5: Reuse the oracles you already have

The hard part of a test is knowing the right answer to check against. That answer-source is called an oracle. When the code already gives you one, the test almost writes itself, so look for these before writing checks by hand:

  • Round-trip: decode(encode(x)) == x, free for any serializer.
  • Idempotence: f(f(x)) == f(x), free for anything meant to be safe to repeat.
  • Invariance: a rule that must still hold after the operation, such as a preserved total or a still-valid schema.
  • A reference: the slow version of a function, kept as the oracle its optimized replacement is diffed against.
  • A decision table for a handful of boolean inputs, among the cheapest formal methods and the one most teams overlook (Wayne 2018)4.
  • A contract: a precondition is already a property, checkable against generated inputs (Wayne 2017)5.

Each of these holds for every input, not just the few cases a hand-written test enumerates.

Step 6: Locate a method for each remaining failure mode

For the failures where you have no oracle yet, match a method to the situation. The methods catalog lists every method by the kind of input it takes, the kind of oracle it uses, and the guarantee a pass earns; the axes explain those keys. The common situations:

  • a reference implementation on hand points to differential testing (a reimplementation against the incumbent; a SIMD kernel against the scalar version it replaced);
  • several independent implementations, or a shared spec suite, give an oracle from agreement (conformance suites, browser reftests, emulator test ROMs);
  • a mature domain with untrusted input points to fuzzing, a free crash-oracle that needs no reference (a parser harness is a few dozen lines on OSS-Fuzz);
  • concurrency over a deterministic core points to simulation testing, and a protocol design to TLA+;
  • heavy state and I/O, as in a typical web app, points to running the code against real dependencies (a real database, real services) instead of stand-ins, so a test covers the wiring and not just the logic;
  • a seam between parts of different owner or guarantee points to a contract or schema and boundary validation;
  • a large configuration or feature-flag space points to combinatorial (pairwise) testing;
  • a performance target points to profiling, microbenchmarking, and load and stress testing;
  • output that is judged, not computed (a ranking, a translation, generated media) points to the soft regime: eval-set statistics, metamorphic relations, comparison against the prior version, and human judgment on samples. Here the guarantee caps at empirical.

Step 7: Combine methods as the effect grows

No single method covers code that touches a lot, and the more a unit touches, the more ways it can fail. The code's effect decides which methods to combine:

As the effect grows Add these methods What they buy
Pure logic (no effect) Examples, property tests, mutation testing Property tests break the regression ceiling; mutation testing measures whether the examples detect anything.
Local I/O and state + fault injection Exercises the failure modes a stateful, side-effecting boundary opens up.
Network and real dependencies + metamorphic or differential oracles, load and stress testing, chaos testing Oracles reach the invariants enumeration cannot; load and stress testing find where throughput and latency break down; chaos testing surfaces the failures only real services produce.
Production, unbounded nondeterminism + whole-system smoke tests, monitoring, canary or shadow traffic A thin top layer that catches what slipped through, in the running system.

The levels are not a strict ladder, since effect is a set of tags rather than a sequence, but the progression is the common one.

Step 8: Measure, then repeat until satisfied

Once a real suite exists, check that it works: mutation testing tells you whether the tests catch faults, and coverage is the gap-finder for a diff. Then loop back to step 6 for the failure modes still worth covering. Two rules keep the loop honest:

  • Always take the by-construction and near-free wins (steps 3 and 4), even on a minor dimension.
  • Never leave a top-priority failure mode uncovered just because its method is expensive.

Stop when the next method would cost more than the failure it prevents. Base that on this part's stakes, not on what similar projects happen to do. Write down what you leave out as a known gap and stage it for when the part grows, its stakes rise, or a bug class recurs.

Scope

These steps cover the fit between methods and a part, not the choice of substrate. The language, framework, and architecture behind step 3 also weigh familiarity, the existing stack, and risk appetite; the steps inform that choice but do not make it. A deterministic core makes simulation cheap. Where step 0 names the data rather than the program, the dimensions and the methods are the same ones; what changes is the vocabulary a data team will recognize them under (ISO/IEC 25012).

Referenced by

References

  1. SQLite Development Team. 2025. How SQLite Is Tested. https://www.sqlite.org/testing.html. ↩

  2. RTCA. 2011a. DO-178C: Software Considerations in Airborne Systems and Equipment Certification. https://www.rtca.org/do-178/. ↩

  3. IEC. 2006. IEC 62304: Medical Device Software — Software Life Cycle Processes. https://webstore.iec.ch/en/publication/6792. ↩

  4. Wayne, Hillel. 2018. Decision Tables. https://www.hillelwayne.com/post/decision-tables/. ↩

  5. Wayne, Hillel. 2017. Introduction to Contract Programming. https://www.hillelwayne.com/post/contracts/. ↩