Skip to content

Software Quality

Change-point detection

Change-point detection reads the whole series of a metric measured over many commits and locates the point where its distribution shifted. That point is the commit that actually moved p99 latency, throughput, build time, or a model-quality score, separated from the noise around it. The series is necessary because a single run only reveals the current build's performance, and comparing two adjacent runs yields almost no signal — the metric is noisy, and the difference between any two builds is mostly variance.

The decision is a statistical test over a distribution — but unlike statistical / sampling testing, which samples one system to reach a verdict now, change-point detection reads history. It consumes the series that a microbenchmark or a load test produces and turns a drifting metric into a dated, attributable event. That is how a project keeps a performance or quality metric from eroding silently over hundreds of commits.

What it catches

  • The regressing commit, not the regressing run. Over a year of noisy benchmark history, the one change that shifted the distribution — found without a per-commit threshold that would fire constantly on variance.
  • Slow drift below a single-step threshold. A metric that degrades 0.3% a week never trips a "5% slower than last build" gate but is unmistakable as a level shift in the series.
  • Improvements and their disappearance. The same machinery flags when an optimization landed and when a later change quietly undid it.

What it does not do is prove causation or judge a single build in isolation. It also assumes the series is comparable point to point — same hardware, same input — or environmental drift masquerades as a change point.

Tools

  • airspeed velocity — runs a benchmark suite across a project's commit history and tracks each metric over time, surfacing regressions as the series moves; the benchmark-history runner used across scientific-Python projects (NumPy, SciPy, Astropy).
  • ruptures — a general-purpose Python change-point library (multiple cost functions and search algorithms); the building block when the series comes from somewhere other than a benchmark runner.

When to use, when not

Use:

  • When a performance or quality metric is tracked in CI over many commits and a shift must be attributed to a specific change rather than chased as noise.
  • When the per-build noise is large enough that comparing adjacent runs gives false alarms — the regime where a level-shift test beats a fixed threshold.
  • As the analysis layer on top of a microbenchmark or load-test suite, not a replacement for it.

Don't:

  • For a one-off "is this build slower?" question. That is a direct comparison or a microbenchmark, not a series analysis.
  • When the series is not comparable across points — the hardware, input, or environment drifted — without normalizing first; the detector will attribute environmental drift to a commit.
  • As proof of cause. The change point names where; the regression's why is found by reading the diff and profiling at that commit.

Evidence

  • MongoDB's CI replaced its threshold-based performance-regression system with change-point detection over the benchmark series and reported it as a measurable improvement — fewer false alarms and reliable attribution of a regression to its commit (Daly et al. 2020)1.

No controlled study measures change-point detection against a fixed per-step threshold; the evidence is a single industrial record. Its value for a noisy longitudinal metric is close to definitional.

Per-commit signals

Both index by commit, but read different things. Git-history hotspots read git log metadata (churn weighted by bug-fix coupling) to rank files by defect risk, before any code runs. Change-point detection reads a measured metric such as p99 latency across builds and finds the commit where it shifted.

Classification

  • Quality dimensions: Maintainability, Performance (attributes a shift in a performance series (p99, throughput) to the commit that caused it; the same machinery serves build time or model quality).
  • Area: Tracking a noisy metric over a project's history — benchmark suites in CI, build/test time, model-quality scores — to locate the commit where the distribution shifted.
  • Guarantee: Empirical — it reports the most likely shift points in the measured series at a chosen confidence, not a proof that a commit caused a regression.

Referenced by

References

  1. Daly, David, William Brown, Henrik Ingo, Jim O'Leary, and David Bradford. 2020. The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System. https://doi.org/10.48550/arXiv.2003.00584. ↩