Skip to content

Software Quality

Statistical and sampling testing

Statistical and sampling testing runs the system many times and judges the outcomes together, because one run of a varying system settles nothing. The property that matters belongs to the distribution: a chess engine is "5 Elo stronger"; an LLM's pass-rate "moved 2 points"; a stochastic protocol "violates its deadline with probability under 0.01."

This is the method that reaches the unbounded tier of the nondeterminism ladder: variation in a real system you can neither remove, seed, nor model — only sample. (When the randomness is modelable — a known stochastic model whose transitions you can simulate — the method is statistical model checking.) Its oracle is the probabilistic one — a statistical decision over the trials, which is what distinguishes it from running an example or property test n times and reading the spread by eye.

What it catches

  • Small or noisy regressions and improvements. A change worth a few Elo or a couple of points of accuracy — real but below the noise floor of a single run. The statistical test separates signal from variance; a threshold on one run cannot.
  • Win-rate and pass-rate movements. "Version B beats version A more often than chance" — an A/B claim with bounded false-accept and false-reject rates, not a single match or a single eval pass.
  • Distributional safety of a real stochastic system. "The measured rate at which this nondeterministic service violates its bound exceeds the threshold" — decided by repeated trials plus a hypothesis test.

It does not give a per-run correctness verdict. It judges the aggregate, so it needs a base signal per trial (a win/draw/loss, a pass/fail, a measured outcome) — a per-run oracle supplies what each trial means. It also assumes the trials are roughly independent and identically distributed; when they are not, the test's error guarantees do not hold.

How the decision is made

  • Fixed-sample test. Size N in advance via power analysis, run all N, then apply a two-proportion test, Welch's t-test (Welch 1947)1, or Mann–Whitney (Mann and Whitney 1947)2; bootstrap (Efron 1979)3 for a percentile statistic.
  • Sequential test. Accumulate evidence and stop the moment it crosses an accept/reject boundary at chosen error rates (α, β). The sequential probability ratio test (Wald 1945)4 does this for a binary outcome; the generalized SPRT (Li et al. 2014)5 for multi-outcome trials. Far fewer trials than a fixed-sample test for the same guarantees.

When to use, when not

Use:

  • When the quantity under test is statistical — engine strength, eval pass-rate, the rate a real stochastic system violates a property. One run cannot answer the question by construction.
  • When single-run tests flake because the behavior is nondeterministic and the flake is the signal, not a defect to suppress.
  • When trials are expensive — the sequential tests stop as early as the evidence allows.

Don't:

  • When one run is a verdict. Deterministic code is settled by an example or property test at a fraction of the cost; a statistical test there measures only noise.
  • When the trials are not independent or identically distributed — autocorrelated series, drifting environments — without accounting for it; the error guarantees silently break.
  • As a substitute for a per-trial correctness oracle. The statistical layer reports how the distribution behaves, not whether any single output was right.

The same statistical machinery serves adjacent problems with different objects and quality dimensions:

  • Microbenchmarking — a single version's performance reported as a distribution, not a verdict on which version is better.
  • Change-point detection — a metric's history across commits, locating where the distribution shifted.

Evidence

  • Stockfish/Fishtest decides every accepted patch by GSPRT (The Stockfish Project 2026)6.
  • LLM evaluation is converging on the same statistical discipline — reporting error bars and significance on a pass-rate rather than a bare number (Miller 2024)7.

The method's value is close to definitional — a statistical quantity needs a statistical verdict; a single sample cannot estimate a distribution — with the sequential-testing literature (Wald 1945; Li et al. 2014)4 5 and the Fishtest record as the practical evidence.

Looks alike — Statistical testing vs statistical model checking

Both sample many runs and make a statistical decision, which makes them look like the same method — but they sample different things. Statistical model checking samples a model whose probabilities you know and can simulate; statistical testing samples a real system whose randomness you can't model, only observe.

Classification

  • Quality dimensions: Functionality, Reliability (estimates failure- or deadline-violation probability over the outcome distribution).
  • Area: Game-engine and ML-model patch evaluation, LLM eval gates, stochastic real-system verification; any code whose verdict needs many noisy trials of a real system aggregated.
  • Guarantee: Empirical — a verdict with bounded false-accept / false-reject rates over the sampled distribution, not a proof.

Referenced by

References

  1. Welch, B. L. 1947. "The Generalization of 'Student's' Problem when Several Different Population Variances are Involved." Biometrika 34 (1–2): 28–35. https://doi.org/10.2307/2332510. ↩

  2. Mann, H. B., and D. R. Whitney. 1947. "On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other." Annals of Mathematical Statistics 18 (1): 50–60. https://doi.org/10.1214/aoms/1177730491. ↩

  3. Efron, B. 1979. "Bootstrap Methods: Another Look at the Jackknife." Annals of Statistics 7 (1): 1–26. https://doi.org/10.1214/aos/1176344552. ↩

  4. Wald, Abraham. 1945. "Sequential Tests of Statistical Hypotheses." Annals of Mathematical Statistics 16 (2): 117–86. https://doi.org/10.1214/aoms/1177731118. ↩↩

  5. Li, Xiaoou, Jingchen Liu, and Zhiliang Ying. 2014. "Generalized Sequential Probability Ratio Test for Separate Families of Hypotheses." Sequential Analysis 33 (4): 539–63. https://doi.org/10.1080/07474946.2014.961861. ↩↩

  6. The Stockfish Project. 2026. Statistical Methods and Algorithms in Fishtest. https://official-stockfish.github.io/docs/fishtest-wiki/Fishtest-Mathematics.html. ↩

  7. Miller, Evan. 2024. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. https://doi.org/10.48550/arXiv.2411.00640. ↩