# Statistical and sampling testing Statistical and sampling testing runs the system many times and judges the outcomes together, because one run of a varying system settles nothing. The property that matters belongs to the *distribution*: a chess engine is "5 Elo stronger"; an LLM's pass-rate "moved 2 points"; a stochastic protocol "violates its deadline with probability under 0.01." This is the method that reaches the **unbounded** tier of the [nondeterminism ladder](https://quality.stereobooster.com/effect.md): variation in a *real* system you can neither remove, seed, nor model — only sample. (When the randomness *is* modelable — a known stochastic model whose transitions you can simulate — the method is [statistical model checking](https://quality.stereobooster.com/probabilistic-model-checking.md).) Its oracle is the probabilistic one — a statistical decision over the trials, which is what distinguishes it from running an [example](https://quality.stereobooster.com/example-tests.md) or [property](https://quality.stereobooster.com/property-based-testing.md) test *n* times and reading the spread by eye. ## What it catches - **Small or noisy regressions and improvements.** A change worth a few Elo or a couple of points of accuracy — real but below the noise floor of a single run. The statistical test separates signal from variance; a threshold on one run cannot. - **Win-rate and pass-rate movements.** "Version B beats version A more often than chance" — an A/B claim with bounded false-accept and false-reject rates, not a single match or a single eval pass. - **Distributional safety of a real stochastic system.** "The measured rate at which this nondeterministic service violates its bound exceeds the threshold" — decided by repeated trials plus a hypothesis test. It does **not** give a *per-run* correctness verdict. It judges the aggregate, so it needs a base signal per trial (a win/draw/loss, a pass/fail, a measured outcome) — a per-run oracle supplies *what* each trial means. It also assumes the trials are roughly independent and identically distributed; when they are not, the test's error guarantees do not hold. ## How the decision is made - **Fixed-sample test.** Size *N* in advance via power analysis, run all *N*, then apply a two-proportion test, Welch's t-test (Welch 1947)[^welch1947], or Mann–Whitney (Mann and Whitney 1947)[^mann1947]; bootstrap (Efron 1979)[^efron1979] for a percentile statistic. - **Sequential test.** Accumulate evidence and stop the moment it crosses an accept/reject boundary at chosen error rates (α, β). The **sequential probability ratio test** (Wald 1945)[^wald1945] does this for a binary outcome; the **generalized SPRT** (Li et al. 2014)[^li2014] for multi-outcome trials. Far fewer trials than a fixed-sample test for the same guarantees. ## When to use, when not **Use:** - When the quantity under test is statistical — engine strength, eval pass-rate, the rate a real stochastic system violates a property. One run cannot answer the question by construction. - When single-run tests *flake* because the behavior is nondeterministic and the flake is the signal, not a defect to suppress. - When trials are expensive — the sequential tests stop as early as the evidence allows. **Don't:** - When one run *is* a verdict. Deterministic code is settled by an [example](https://quality.stereobooster.com/example-tests.md) or [property](https://quality.stereobooster.com/property-based-testing.md) test at a fraction of the cost; a statistical test there measures only noise. - When the trials are not independent or identically distributed — autocorrelated series, drifting environments — without accounting for it; the error guarantees silently break. - As a substitute for a per-trial correctness oracle. The statistical layer reports *how the distribution behaves*, not whether any single output was right. ## Related methods The same statistical machinery serves adjacent problems with different objects and quality dimensions: - [Microbenchmarking](https://quality.stereobooster.com/microbenchmarking.md) — a *single version's* performance reported as a distribution, not a verdict on which version is better. - [Change-point detection](https://quality.stereobooster.com/change-point-detection.md) — a metric's *history* across commits, locating where the distribution shifted. ## Evidence - **Stockfish/Fishtest** decides every accepted patch by GSPRT (The Stockfish Project 2026)[^stockfish2026]. - **LLM evaluation** is converging on the same statistical discipline — reporting error bars and significance on a pass-rate rather than a bare number (Miller 2024)[^miller2024]. The method's *value* is close to definitional — a statistical quantity needs a statistical verdict; a single sample cannot estimate a distribution — with the sequential-testing literature (Wald 1945; Li et al. 2014)[^wald1945] [^li2014] and the Fishtest record as the practical evidence. ## Related **Looks alike — Statistical testing vs statistical model checking** Both sample many runs and make a statistical decision, which makes them look like the same method — but they sample different things. [Statistical model checking](https://quality.stereobooster.com/probabilistic-model-checking.md) samples a *model* whose probabilities you know and can simulate; statistical testing samples a *real system* whose randomness you can't model, only observe. ## Classification - **Quality dimensions:** Functionality, Reliability (estimates failure- or deadline-violation probability over the outcome distribution). - **Area:** Game-engine and ML-model patch evaluation, LLM eval gates, stochastic real-system verification; any code whose verdict needs many noisy trials of a real system aggregated. - **Guarantee:** Empirical — a verdict with bounded false-accept / false-reject rates over the sampled distribution, not a proof. ## Referenced by - [Quality dimensions](https://quality.stereobooster.com/quality-dimensions.md) · Quality dimensions - [Effect scope](https://quality.stereobooster.com/effect.md) · The axes - [Oracle](https://quality.stereobooster.com/oracle.md) · The axes - [Change-point detection](https://quality.stereobooster.com/change-point-detection.md) · Methods - [Probabilistic model checking](https://quality.stereobooster.com/probabilistic-model-checking.md) · Methods - [Snapshot and approval testing](https://quality.stereobooster.com/snapshot-testing.md) · Methods - [How AI fits into software quality](https://quality.stereobooster.com/ai.md) · AI ## References [^welch1947]: Welch, B. L. 1947. "[The Generalization of 'Student's' Problem when Several Different Population Variances are Involved](https://www.jstor.org/stable/2332510)." *Biometrika* 34 (1–2): 28–35. . [^mann1947]: Mann, H. B., and D. R. Whitney. 1947. "[On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other](https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2Faoms%2F1177730491)." *Annals of Mathematical Statistics* 18 (1): 50–60. . [^efron1979]: Efron, B. 1979. "[Bootstrap Methods: Another Look at the Jackknife](https://sites.stat.washington.edu/courses/stat527/s14/readings/ann_stat1979.pdf)." *Annals of Statistics* 7 (1): 1–26. . [^wald1945]: Wald, Abraham. 1945. "[Sequential Tests of Statistical Hypotheses](https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-16/issue-2/Sequential-Tests-of-Statistical-Hypotheses/10.1214/aoms/1177731118.pdf)." *Annals of Mathematical Statistics* 16 (2): 117–86. . [^li2014]: Li, Xiaoou, Jingchen Liu, and Zhiliang Ying. 2014. "[Generalized Sequential Probability Ratio Test for Separate Families of Hypotheses](https://pmc.ncbi.nlm.nih.gov/articles/PMC4941833/)." *Sequential Analysis* 33 (4): 539–63. . [^stockfish2026]: The Stockfish Project. 2026. *[Statistical Methods and Algorithms in Fishtest](https://official-stockfish.github.io/docs/fishtest-wiki/Fishtest-Mathematics.html)*. . [^miller2024]: Miller, Evan. 2024. *[Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://arxiv.org/pdf/2411.00640)*. . ## Acronyms - GSPRT — generalized sequential probability ratio test - SPRT — sequential probability ratio test