# Regression-suite management

Regression-suite management takes the *suite* as its object rather than the
code, and decides which of its tests to run, in what order, and which to
drop. A regression suite grows monotonically: every fixed bug leaves a test
behind, every feature adds more. Left alone it eventually costs more to run
than the change it guards is worth, and on a large codebase running all of
it on every commit becomes impossible. [Measuring test-suite
effectiveness](https://quality.stereobooster.com/measuring-test-effectiveness.md) shares that object but asks
a different question: *how good are the tests*.

Measuring **produces** the signals (a coverage profile, a mutation score,
a fault history); managing **consumes** them to make a scheduling
decision. A coverage-based prioritizer orders tests by the coverage each
one adds; a history-based prioritizer runs the tests that failed recently
first. So these methods do not judge a run and do not verify the code.
They reorganize an existing suite to bound the cost and latency of
regression testing.

## The problems

The canonical division (Yoo and Harman 2012)[^yoo2012] separates the problem by what a
technique changes and what it risks:

- **[Minimization](https://quality.stereobooster.com/test-suite-minimization.md)** permanently *deletes*
  tests judged redundant against a criterion. It is the most aggressive
  and the most dangerous, because a test redundant *for coverage* may be
  the only one that detects a given fault.
- **[Selection](https://quality.stereobooster.com/regression-test-selection.md)** *skips* the tests a
  specific change cannot affect, and only for that run. Its defining
  property is *safety*.
- **[Prioritization](https://quality.stereobooster.com/test-case-prioritization.md)** *reorders* the suite
  to run likely failures first. It deletes and skips nothing, so it
  carries no correctness risk; its only lever is *when* the failure
  surfaces.

Selection and prioritization compose: prioritization orders whatever
subset selection leaves.

## The properties that matter

**Safety** (for selection). A regression test selection technique is
*safe* if it selects every test in the suite that can expose a fault in
the changed program (Rothermel and Harrold 1997)[^rothermel1997]. An unsafe technique trades that
guarantee for a smaller, cheaper subset. Safety is a claim about the
*selection*, not about the code, so it does not appear on the guarantee
axis. It is the same shape of promise: no false negatives, relative to
the change model the technique uses.

**Rate of fault detection** (for prioritization). Since prioritization
changes only order, its benefit is measured over the *sequence* in which
the tests run. The standard metric is **APFD**, the average percentage of
faults detected (Rothermel et al. 2001)[^rothermel2001]: the area under the curve of "faults
found" as a function of "fraction of the suite run". An ordering that
finds all faults in the first 10% of the suite scores near 100; one that
finds them only in the last 10% scores near 0.

## Referenced by

- [Regression test selection](https://quality.stereobooster.com/regression-test-selection.md) · Methods
- [Test case prioritization](https://quality.stereobooster.com/test-case-prioritization.md) · Methods
- [Test suite minimization](https://quality.stereobooster.com/test-suite-minimization.md) · Methods
- [Testing GUI and mobile applications](https://quality.stereobooster.com/testing-gui-and-mobile-apps.md) · Methods
- [The test suite as an object](https://quality.stereobooster.com/test-suite.md) · Methods

## References

[^yoo2012]: Yoo, Shin, and Mark Harman. 2012. "[Regression Testing Minimization, Selection and Prioritization: A Survey](https://coinse.github.io/publications/pdfs/Yoo2010fk.pdf)." *Software Testing, Verification and Reliability* 22 (2): 67–120. <https://doi.org/10.1002/stvr.430>.
[^rothermel1997]: Rothermel, Gregg, and Mary Jean Harrold. 1997. "[A Safe, Efficient Regression Test Selection Technique](https://www.cs.purdue.edu/homes/xyzhang/spring07/Papers/p173-rothermel.pdf)." *ACM Transactions on Software Engineering and Methodology* 6 (2): 173–210. <https://doi.org/10.1145/248233.248262>.
[^rothermel2001]: Rothermel, Gregg, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 2001. "[Prioritizing Test Cases for Regression Testing](https://niplav.site/doc/cs/reduction/prioritizing_test_cases_for_regression_testing_rothermel_et_al_2001.pdf)." *IEEE Transactions on Software Engineering* 27 (10): 929–48. <https://doi.org/10.1109/32.962562>.

## Acronyms

- APFD — average percentage of faults detected