Skip to content

Software Quality

Example tests

An example test pairs a hand-picked input with an assertion about what the code does with it. The assertion is usually an exact expected value, but equally a property the output must satisfy, a round-trip relation, or merely that nothing crashed. It is the foundational verification technique in most codebases: the everyday categories of automated test are all example tests, separated only by how much of the system they exercise.

Example tests are the right tool when the relevant cases are small in number, enumerable, and known to the author. Past that range the author cannot write the cases out, and property-based testing or fuzzing generates them instead.

What it catches

  • Regressions of fixed bugs. Once an example test captures the input that caused the bug, the same case cannot reach production silently again. Most regression suites are exactly this — a record of the failures already found.
  • Golden-path correctness. The common, expected use of the feature works as specified.
  • Specified edge cases. The empty list, the zero amount and the max capacity behave as the author wrote them down.
  • Behavior the spec enumerates. A finite set of declared cases (state transitions, status codes, mode toggles) maps naturally to a finite set of example tests, one per case.
  • Documented examples. Test as documentation: the test is the worked example a future reader needs.

What example tests structurally cannot catch

Example tests guarantee the absence of known bugs, not the absence of bugs. A regression test needs no more than that: the bug it pins is already known.

Tools

Every mainstream language has stable example-test infrastructure. The runners named for each language are common defaults, not a complete set.

Most of these support: fixtures, parameterized tests (one example body × multiple input rows), parallel execution, and snapshot/approval captures.

When to use, when not

Use example tests:

  • For every bug fix — pin the case as a regression.
  • For specified, finite behavior — a state machine with a handful of transitions; a function with three error codes.
  • As golden-path system tests — one happy-path scenario per high-value flow, exercised through the assembled system.
  • For smoke checks at a deploy boundary — fast, heuristic-tier, deliberately limited.
  • As documentation. A failing test is a precise bug report; a passing test is an executable example.
  • When other methods would be overkill. A configuration loader with three valid shapes and two error cases doesn't need a property-based test; three example tests are correct.

Don't lean only on example tests when:

  • The input space is rich and irregular — parsers, encoders, numerical code, ML, anything that takes "data with structure". Combine with property-based testing, fuzzing, metamorphic testing, or differential testing. Example tests still belong in the suite for the cases that specifically matter; they are not the primary input.
  • The code is concurrent or distributed. State-space explosion makes hand-enumeration insufficient; see deterministic simulation testing and model checking.
  • The boundary is security-sensitive. Pair with fuzzing, threat modeling, and static analysis.
  • AI generated the code. AI's bugs are plausibly-shaped, locally-coherent, subtly wrong — exactly the kind that hides between AI-generated code and AI-generated example tests written from the same intuition. The other methods don't share that intuition.

Evidence

There is no controlled study of example tests as a technique: example-based testing is the baseline other methods are measured against. The case for it is definitional rather than empirical. A suite of N example tests establishes only that the code behaves correctly on those N inputs.

Classification

  • Quality dimensions: Functionality, Maintainability (as a regression / snapshot refactor-safety net).
  • Area: Regression prevention, golden-path coverage, bug-fix verification, fast-feedback during development; any code with small, enumerable behavior the author can pin down by hand.
  • Guarantee: Empirical — the cases tried passed.

Referenced by