# Test case prioritization

Prioritization *orders* the whole suite so that the tests most likely to reveal a
fault run first. It removes nothing and skips nothing, so it carries no
correctness risk: run long enough and every test still executes. Its only lever
is *when* you find out, which makes it the lowest-risk of the
[regression-suite management](https://quality.stereobooster.com/regression-management.md) techniques.

## What it does

On a suite that takes an hour, an ordering that surfaces the first failure two
minutes in rather than fifty gives the developer the same information
forty-eight minutes earlier, with the total run time unchanged. The benefit is
measured by [APFD](https://quality.stereobooster.com/regression-management.md#the-properties-that-matter), the rate
at which the ordering accumulates fault detection.

## Coverage- and history-based ordering

- **Coverage-based.** Order tests by the coverage each one contributes. *Total
  greedy* sorts tests by the coverage each achieves on its own; *additional
  greedy* repeatedly picks the test that adds the most not-yet-covered code,
  recomputing after each pick. Coverage-based ordering needs instrumentation and
  a stored coverage profile per test, so it costs something to maintain.
- **History-based.** Order by past behavior: tests that failed recently, that
  fail often, or that have not run in a while go first. It needs no
  instrumentation, only the CI system's own pass/fail log. That is what makes it
  applicable under continuous integration, where coverage-based techniques are
  hard to apply: they assume instrumentation and a discrete, complete test suite
  (Elbaum et al. 2014)[^elbaum2014].

## Tools

No dedicated tool is required: a history-based ordering is a few lines of
scheduling code over the CI system's own pass/fail log, ordering by last-failure
time or failure frequency. What build tools and hosted CI ship instead is
dependency-based [selection](https://quality.stereobooster.com/regression-test-selection.md), which skips tests
rather than reordering them.

## When to use, when not

**Use it almost always.** An ordering that turns out badly costs only the time the
run would have taken anyway.

**Don't reach for coverage-based ordering when history-based will do.** The extra
instrumentation rarely pays for itself in a CI setting.

## Evidence

- **Prioritization improves the fault-detection rate.** In the family of
  empirical studies that established the area, every technique considered
  improved that rate overall (Elbaum et al. 2002)[^elbaum2002]. Feedback alone does not decide the
  ranking: *additional* statement coverage came last among the statement-level
  heuristics compared, significantly worse than *total* coverage, while the
  technique that ranked first combined additional coverage with an estimate of
  each test's fault-exposing potential. The gains are real but bounded, and the
  ranking of heuristics is context-dependent rather than universal.
- **Ordering by failure history shortens the time to a failure report**, measured
  against no prioritization on a Google dataset of millions of test-suite
  executions (Elbaum et al. 2014)[^elbaum2014].

## Classification

- **Quality dimensions:** Maintainability (shortens feedback latency by running likely-failing tests first; no correctness risk) — orders an existing suite so likely failures surface first; it schedules tests, it does not run or verify them, so it sits off the code-verification axes.
- **Area:** Long regression suites where the wait for the first failure matters: continuous-integration pipelines and end-to-end suites.

## Referenced by

- [Regression-suite management](https://quality.stereobooster.com/regression-management.md) · Methods
- [The test suite as an object](https://quality.stereobooster.com/test-suite.md) · Methods

## References

[^elbaum2014]: Elbaum, Sebastian, Gregg Rothermel, and John Penix. 2014. "[Techniques for Improving Regression Testing in Continuous Integration Development Environments](https://cs.uwaterloo.ca/~m2nagapp/courses/CS846/1189/papers/elbaum_fse14.pdf)." *Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014)*, 235–45. <https://doi.org/10.1145/2635868.2635910>.
[^elbaum2002]: Elbaum, Sebastian, Alexey G. Malishevsky, and Gregg Rothermel. 2002. "[Test Case Prioritization: A Family of Empirical Studies](https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1018&context=csearticles)." *IEEE Transactions on Software Engineering* 28 (2): 159–82. <https://doi.org/10.1109/32.988497>.

## Acronyms

- APFD — average percentage of faults detected