# Change-point detection Change-point detection reads the *whole series* of a metric measured over many commits and locates the point where its distribution shifted. That point is the commit that actually moved p99 latency, throughput, build time, or a model-quality score, separated from the noise around it. The series is necessary because a single run only reveals the current build's performance, and comparing two adjacent runs yields almost no signal — the metric is noisy, and the difference between any two builds is mostly variance. The decision is a statistical test over a distribution — but unlike [statistical / sampling testing](https://quality.stereobooster.com/statistical-testing.md), which samples *one* system to reach a verdict now, change-point detection reads *history*. It consumes the series that a [microbenchmark](https://quality.stereobooster.com/microbenchmarking.md) or a load test produces and turns a drifting metric into a dated, attributable event. That is how a project keeps a performance or quality metric from eroding silently over hundreds of commits. ## What it catches - **The regressing commit, not the regressing run.** Over a year of noisy benchmark history, the one change that shifted the distribution — found without a per-commit threshold that would fire constantly on variance. - **Slow drift below a single-step threshold.** A metric that degrades 0.3% a week never trips a "5% slower than last build" gate but is unmistakable as a level shift in the series. - **Improvements and their disappearance.** The same machinery flags when an optimization landed and when a later change quietly undid it. What it does **not** do is prove causation or judge a single build in isolation. It also assumes the series is comparable point to point — same hardware, same input — or environmental drift masquerades as a change point. ## Tools - **[airspeed velocity](https://github.com/airspeed-velocity/asv)** — runs a benchmark suite across a project's commit history and tracks each metric over time, surfacing regressions as the series moves; the benchmark-history runner used across scientific-Python projects (NumPy, SciPy, Astropy). - **[ruptures](https://centre-borelli.github.io/ruptures-docs/)** — a general-purpose Python change-point library (multiple cost functions and search algorithms); the building block when the series comes from somewhere other than a benchmark runner. ## When to use, when not **Use:** - When a performance or quality metric is tracked in CI over many commits and a shift must be attributed to a specific change rather than chased as noise. - When the per-build noise is large enough that comparing adjacent runs gives false alarms — the regime where a level-shift test beats a fixed threshold. - As the analysis layer on top of a [microbenchmark](https://quality.stereobooster.com/microbenchmarking.md) or load-test suite, not a replacement for it. **Don't:** - For a one-off "is this build slower?" question. That is a direct comparison or a [microbenchmark](https://quality.stereobooster.com/microbenchmarking.md), not a series analysis. - When the series is not comparable across points — the hardware, input, or environment drifted — without normalizing first; the detector will attribute environmental drift to a commit. - As proof of cause. The change point names *where*; the regression's *why* is found by reading the diff and [profiling](https://quality.stereobooster.com/profiling.md) at that commit. ## Evidence - **MongoDB's CI** replaced its threshold-based performance-regression system with change-point detection over the benchmark series and reported it as a measurable improvement — fewer false alarms and reliable attribution of a regression to its commit (Daly et al. 2020)[^daly2020]. No controlled study measures change-point detection against a fixed per-step threshold; the evidence is a single industrial record. Its value for a noisy longitudinal metric is close to definitional. ## Related **Per-commit signals** Both index by commit, but read different things. [Git-history hotspots](https://quality.stereobooster.com/git-hotspots.md) read `git log` metadata (churn weighted by bug-fix coupling) to rank files by defect risk, before any code runs. Change-point detection reads a measured metric such as p99 latency across builds and finds the commit where it shifted. ## Classification - **Quality dimensions:** Maintainability, Performance (attributes a shift in a performance series (p99, throughput) to the commit that caused it; the same machinery serves build time or model quality). - **Area:** Tracking a noisy metric over a project's history — benchmark suites in CI, build/test time, model-quality scores — to locate the commit where the distribution shifted. - **Guarantee:** Empirical — it reports the most likely shift points in the measured series at a chosen confidence, not a proof that a commit caused a regression. ## Referenced by - [Quality dimensions](https://quality.stereobooster.com/quality-dimensions.md) · Quality dimensions - [Microbenchmarking](https://quality.stereobooster.com/microbenchmarking.md) · Methods - [Statistical and sampling testing](https://quality.stereobooster.com/statistical-testing.md) · Methods - [Testing machine-learning systems](https://quality.stereobooster.com/testing-ml-systems.md) · Methods - [How AI fits into software quality](https://quality.stereobooster.com/ai.md) · AI ## References [^daly2020]: Daly, David, William Brown, Henrik Ingo, Jim O'Leary, and David Bradford. 2020. *[The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System](https://arxiv.org/pdf/2003.00584)*. .