Skip to content

Software Quality

A field guide to knowing whether your software works.

The cost shift

AI cuts the cost of producing code, and expectations about delivery speed rise with it — but a change still has to be read, tested, and trusted, and that costs about what it always did. The bottleneck has moved to verification.

This project reviews the ways of finding out whether software works, drawing on the literature and on industry practice.

What this project does

  • Surveys the methods in use. Types, static analysis, tests, runtime checks, formal methods, monitoring: one page per method, each stating what uncertainty it reduces, the tools that implement it, and the evidence behind it. See Methods.
  • Organizes them with a few classifications. These are not new methods, and mostly not new vocabulary; where the field already has a standard taxonomy, the classifications reuse it and assemble the pieces into one cross-method frame. The payoff is that relationships become visible: differential, metamorphic, and property-based testing differ mainly in their oracle, not their inputs; coverage and exhaustiveness are points on the guarantee scale, signals of reach rather than methods in their own right. See the axes.
  • Examines how AI shifts these practices. Organized by AI's relationship to the oracle — where AI is the oracle, consumes one, or writes one. See How AI fits into software quality.

Recommendations carry their evidence, or a plain note where it's thin; practices that rest on repetition rather than data are named as folklore.

The dimensions of quality

Quality is several distinct things, not a single scale (Garvin 1987)1 — and each needs its own verification. Five dimensions are developed here: Functionality, Security, Reliability, Performance, Maintainability. Each aligns with an ISO/IEC 25010 product-quality characteristic (ISO/IEC 2011)2 that carries a distinct verification method. Each has its own failure modes, erosion signals, and methods in order of adoption: see Quality dimensions for the per-dimension breakdown. Usability (likewise a 25010 characteristic) and Aesthetics are named but left to their own mature fields.

How to read this

The content answers two questions — what verification methods exist? and which should I reach for, given the quality dimensions I care about? — across these sections:

  • Methods — the verification methods catalog: one page per method, listed the same way throughout. The master table lists every method on a row, and the axes defines the classification methodology.
  • Quality dimensions — which methods apply to a given quality dimension, and in what order.
  • Choosing methods — a heuristic for tying dimensions, axes, and methods together: given a part's stakes and the oracles, guarantee, and tooling on hand, which methods to reach for, in what order, and which to stage for later.
  • AI — AI's effect on each method, with the open-source tools and studies that back each cell.
  • Conventional — the testing vocabulary most developers already use, reconciled with the rest: unit / integration / e2e mapped onto the axes, the pyramid critique, and the two category errors (tests are not all verification; hygiene is not verification).
  • Recipes — a small, opinionated appendix of practical configurations and snippets.
  • Glossary — project-wide vocabulary (oracle, regression ceiling, by construction, folklore as an evidence tier).
  • About — how the content is checked, what that does not guarantee, and who writes it.

References

  1. Garvin, David A. 1987. "Competing on the Eight Dimensions of Quality." Harvard Business Review 65 (6): 101–9. https://hbr.org/1987/11/competing-on-the-eight-dimensions-of-quality. ↩

  2. ISO/IEC. 2011. ISO/IEC 25010:2011: Systems and software engineering — SQuaRE — System and software quality models. International Organization for Standardization. https://www.iso.org/standard/35733.html. ↩