Skip to content

Software Quality

Regression-suite management

Regression-suite management takes the suite as its object rather than the code, and decides which of its tests to run, in what order, and which to drop. A regression suite grows monotonically: every fixed bug leaves a test behind, every feature adds more. Left alone it eventually costs more to run than the change it guards is worth, and on a large codebase running all of it on every commit becomes impossible. Measuring test-suite effectiveness shares that object but asks a different question: how good are the tests.

Measuring produces the signals (a coverage profile, a mutation score, a fault history); managing consumes them to make a scheduling decision. A coverage-based prioritizer orders tests by the coverage each one adds; a history-based prioritizer runs the tests that failed recently first. So these methods do not judge a run and do not verify the code. They reorganize an existing suite to bound the cost and latency of regression testing.

The problems

The canonical division (Yoo and Harman 2012)1 separates the problem by what a technique changes and what it risks:

  • Minimization permanently deletes tests judged redundant against a criterion. It is the most aggressive and the most dangerous, because a test redundant for coverage may be the only one that detects a given fault.
  • Selection skips the tests a specific change cannot affect, and only for that run. Its defining property is safety.
  • Prioritization reorders the suite to run likely failures first. It deletes and skips nothing, so it carries no correctness risk; its only lever is when the failure surfaces.

Selection and prioritization compose: prioritization orders whatever subset selection leaves.

The properties that matter

Safety (for selection). A regression test selection technique is safe if it selects every test in the suite that can expose a fault in the changed program (Rothermel and Harrold 1997)2. An unsafe technique trades that guarantee for a smaller, cheaper subset. Safety is a claim about the selection, not about the code, so it does not appear on the guarantee axis. It is the same shape of promise: no false negatives, relative to the change model the technique uses.

Rate of fault detection (for prioritization). Since prioritization changes only order, its benefit is measured over the sequence in which the tests run. The standard metric is APFD, the average percentage of faults detected (Rothermel et al. 2001)3: the area under the curve of "faults found" as a function of "fraction of the suite run". An ordering that finds all faults in the first 10% of the suite scores near 100; one that finds them only in the last 10% scores near 0.

Referenced by

References

  1. Yoo, Shin, and Mark Harman. 2012. "Regression Testing Minimization, Selection and Prioritization: A Survey." Software Testing, Verification and Reliability 22 (2): 67–120. https://doi.org/10.1002/stvr.430. ↩

  2. Rothermel, Gregg, and Mary Jean Harrold. 1997. "A Safe, Efficient Regression Test Selection Technique." ACM Transactions on Software Engineering and Methodology 6 (2): 173–210. https://doi.org/10.1145/248233.248262. ↩

  3. Rothermel, Gregg, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 2001. "Prioritizing Test Cases for Regression Testing." IEEE Transactions on Software Engineering 27 (10): 929–48. https://doi.org/10.1109/32.962562. ↩