Skip to content

Software Quality

A field guide to knowing whether your software works.

The cost shift

AI cuts the cost of producing code, and expectations about delivery speed rise with it — but a change still has to be read, tested, and trusted. The bottleneck has moved to verification.

This project reviews the ways of finding out whether software works, drawing on the literature and on industry practice.

What this project does

  • Surveys the methods in use. Types, static analysis, tests, runtime checks, formal methods, monitoring: one page per method, each stating what uncertainty it reduces, the tools that implement it, and the evidence behind it. See Methods.
  • Organizes them on a few axes. Where the field already has a standard taxonomy, the axes reuse it and assemble the pieces into one cross-method frame. The payoff is that relationships become visible: differential, metamorphic, and property-based testing differ mainly in their oracle, not their inputs; exhaustiveness is a point on the guarantee scale and coverage a measure of reachability, neither a method in its own right. See the axes.
  • Examines how AI shifts these practices. Organized by AI's relationship to the oracle — where AI is the oracle, consumes one, or writes one. See How AI fits into software quality.

Recommendations carry their evidence, or a plain note where it's thin; practices that rest on repetition rather than data are named as folklore.

The dimensions of quality

Quality is several distinct things, not a single scale (Garvin 1987)1 — and each needs its own verification. Five dimensions are developed here: Functionality, Security, Reliability, Performance, Maintainability. Each aligns with an ISO/IEC 25010 product-quality characteristic (ISO/IEC 2011)2. Usability (likewise a 25010 characteristic) and Aesthetics are named but left to their own mature fields.

How to read this

The content answers two questions — what verification methods exist? and which should I reach for, given the quality dimensions I care about? — across these sections:

  • Methods — the verification methods catalog, whose master table lists every method on a row; the axes page defines the classification methodology.
  • Quality dimensions — which methods apply to a given quality dimension, and in what order.
  • Choosing methods — a heuristic for tying dimensions, axes, and methods together: given a part's stakes and the oracles, guarantee, and tooling on hand, which methods to reach for, in what order, and which to stage for later.
  • AI — AI's effect on each method, with the open-source tools and studies that back each cell.
  • Conventional — the testing vocabulary most developers already use, reconciled with the rest: unit / integration / e2e mapped onto the axes, and the pyramid critique.
  • Recipes — a small, opinionated appendix of practical configurations and snippets.
  • Glossary — project-wide vocabulary (oracle, regression ceiling, by construction, folklore as an evidence tier).
  • About — how the content is checked, what that does not guarantee, and who writes it.

References

  1. Garvin, David A. 1987. "Competing on the Eight Dimensions of Quality." Harvard Business Review 65 (6): 101–9. https://hbr.org/1987/11/competing-on-the-eight-dimensions-of-quality. ↩

  2. ISO/IEC. 2011. ISO/IEC 25010:2011: Systems and software engineering — SQuaRE — System and software quality models. International Organization for Standardization. https://www.iso.org/standard/35733.html. ↩