Skip to content

Software Quality

Functionality

Does the software do what it claims?

Functionality is the quality dimension concerned with whether the software's behavior matches what it claims to do. Most of the testing tradition's methods target it. Two classes of method divide the work: example tests pin the cases written down, and generative methods reach the input space enumeration can't cover. A functionality verification program carries both.

What goes wrong

  • Incorrect output for known inputs. Example regression tests catch these, once written.
  • Incorrect output for inputs nobody thought to test. Example enumeration falls short here by construction; property-based, fuzzing, differential, and metamorphic testing target this gap.
  • Edge cases. Off-by-one, null/empty/overflow, time zones and the boundary of integer ranges are often invisible to spec writers, so the spec doesn't enumerate them.
  • Concurrency and ordering. Race conditions, lost updates and out-of-order delivery are found by TLA+ and deterministic simulation and missed by example tests.
  • API contract drift. Two services agree to a spec and one drifts. Static schema-compatibility checks (OpenAPI diffing, protobuf breaking-change detection) catch syntactic breaks; live integration tests catch semantic ones; consumer-driven contract tests catch semantic breaks without a live counterpart (snapshot testing at the seam).

What success looks like

  • Bug-fix commits include a test. The regression ceiling limits example tests to bugs someone already knows about, so a bug fixed without a test can recur unnoticed.
  • Diff-time signals fire before merge. Coverage, linter, and types on touched files.
  • Property tests cover the input space that can be described. A bug that falls outside that space marks the next gap.
  • The next bug class has a named oracle. For input classes that can't be enumerated, a property, a reference implementation, or a metamorphic relation is the oracle to reach for next.

Referenced by