Skip to content

Software Quality

The test suite as an object

The suite as an object of study: how effective it is, what it costs, and how to keep it healthy.

A test suite is an artifact with its own qualities, and "a good suite" means several different things at once, each with its own measure and its own remedy. Most methods in this catalog reduce uncertainty about the code; the methods and measures here study the suite itself.

How effective is it?

Whether the suite actually catches faults is its effectiveness, which measuring test-suite effectiveness explains through the RIPR chain. Coverage measures only reachability, the chain's first step, so it is a weak proxy. The sharper single measure is mutation testing, which exercises the whole chain rather than just reach.

What does it cost to run?

When a suite grows slow enough to throttle feedback, regression-suite management buys the time back three ways: prioritization reorders so likely failures surface first, selection skips the tests a change cannot affect, and minimization deletes redundant tests outright (the most aggressive, and the one to treat with care).

Is its verdict trustworthy?

An unstable verdict undermines every measure of effectiveness. Flaky tests can be provoked deliberately rather than waited for. The suite-level number is the flake rate: the share of runs whose failure passes on a retry of the same commit, which the maintainability dimension tracks against a target.

Which tests need attention?

The per-test history that steers prioritization also ranks the suite's own problems: the tests that fail flakily, fail most often, or run slowest are the ones worth deflaking, splitting, or deleting first. This is the test-suite counterpart of git-history hotspots for production code, and a test that never fails and covers nothing new is a minimization candidate.

Is it maintainable?

Beyond speed and stability, a suite can be hard to read or brittle to change, and a failure can point cleanly at its cause or not. That last property, diagnosability, is one of the other qualities on which a suite is judged. The test smells catalog names test-design problems, and its claims are largely unmeasured.

Referenced by