# Effect scope **Effect** is what the **code under test** touches — state, I/O, and nondeterminism. It is a property of the code, not of a method, which sets it apart from [input](https://quality.stereobooster.com/input.md), [oracle](https://quality.stereobooster.com/oracle.md), and [guarantee](https://quality.stereobooster.com/guarantee.md). **Effect scope** is the same cut read the other way round: the richest effect a method can verify faithfully. What changes the method is a coarse cut of the code's effect — three dimensions:
- Nondeterminism - **enumerable** (by hand) - **bounded** - **unbounded** - I/O - **local** - **network** (distributed) - **State**
A code's effect is the **set** of tags it carries (for example, `[state, local I/O, bounded ndet]`). Only the dimensions that are genuinely in play get tagged; a pure function carries none, which is exactly why it's trivial to test. Each tag is one more thing a test has to handle. The **magnitude of nondeterminism** is the dominant dimension, so it gets its own tiers; **I/O level** and **state** are coarser. ## The nondeterminism ladder Incidental nondeterminism comes off first. When it isn't part of the behavior under test — a timestamp written into a template, a tie-break RNG — seeding or freezing the clock restores a deterministic test, and most flakiness lives here. When the choice does affect the result the nondeterminism is genuine, and the topmost tier that fits applies: 1. **Enumerate it by hand — when the space is tiny.** Some nondeterminism can't be removed without removing the thing under test: the *interleavings* of two promises or two threads **are** the behavior. Two or three orderings fit in as many explicit cases. 2. **Exhaust or explore it — when it's bounded but beyond hand enumeration.** An interleaving explorer like [loom](https://github.com/tokio-rs/loom) — [systematic concurrency testing](https://quality.stereobooster.com/systematic-concurrency-testing.md) — runs every permitted schedule of the real program; [model checking](https://quality.stereobooster.com/model-checking.md) checks an abstract state machine ([TLA+](https://lamport.azurewebsites.net/tla/tla.html) / [TLC](https://github.com/tlaplus/tlaplus)); [deterministic simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md) seeds *every* source of nondeterminism so a fuzzer can drive a reproducible space (FoundationDB). (Model checking is exhaustive over a *model*; DST samples a *controlled, replayable* space — both reach orderings hand enumeration never would.) 3. **Sample it — when it's unbounded.** Some nondeterminism can be neither seeded nor modeled — there's no closed space to enumerate, only a distribution to sample: a real LLM's outputs, or a statistical quantity no single run can settle. (A network is *not* this case — its losses and reorderings are a closed space TLA+ verifies and DST replays; that's bounded, tier 2.) One run is no longer a verdict: run it *n* times and judge the distribution — a [probabilistic oracle](https://quality.stereobooster.com/oracle.md#probabilistic-a-claim-about-the-distribution). **Stockfish** decides every patch by playing a batch of games against the current version rather than by any single game (The Stockfish Project 2026)[^stockfish2026]; LLM eval suites that gate on a pass-rate do the same. **Unaccounted nondeterminism is exactly a [flaky test](https://quality.stereobooster.com/flaky-tests.md)** — a live source nobody chose a tier for, so the same code passes or fails by luck of the schedule, the clock, or the seed. ## Which method fits Most methods handle the easy combinations; nondeterminism and distribution are what force a specialist: | Method | Scope | | ------ | ----- | | [Example](https://quality.stereobooster.com/example-tests.md) / [property-based](https://quality.stereobooster.com/property-based-testing.md) / [fuzzing](https://quality.stereobooster.com/fuzzing.md) | any effect up to *bounded* nondeterminism — above it one run isn't a verdict, and fuzzing can't triage a crash it can't reproduce | | [Model checking](https://quality.stereobooster.com/model-checking.md) | concurrency + state, abstracted — not faithful real-world `io` | | [Deterministic simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md) | concurrent / distributed, sealed and reproducible | | [Monitoring](https://quality.stereobooster.com/monitoring-and-observability.md) | live `io` / distributed — needs a deployed system | | [Statistical / sampling testing](https://quality.stereobooster.com/statistical-testing.md) | unbounded nondeterminism — no space to model, only a distribution to sample (real LLMs, statistical outcomes) | ## Effect scope: a method's effect ceiling Each method carries an **effect scope**, and selection is mechanical: for code with effect *X*, the candidates are the methods whose scope covers *X*. A method whose scope falls short doesn't fail loudly; it quietly verifies a *reduced* version of the system. Effect scope is **breadth** — *which* effects a method covers, not *how deeply* a run exercises them. Depth is *reachability*: how much of a method's space a run exercises. Chaos engineering covers a *wider* effect scope than deterministic simulation testing yet is the *shallower* method. A method earns a scope tag only when it *controls* the effect space — constructs it, or reasons over all of it — rather than merely observing, measuring, or applying regardless. - **Effect scope doesn't apply to observers or the indifferent.** Several method categories control no effect space: - *Effect-agnostic* — applies to any code regardless of what it touches: [clone detection](https://quality.stereobooster.com/clone-detection.md), a plain [type system](https://quality.stereobooster.com/static-types.md), [code review](https://quality.stereobooster.com/code-review.md). Scope is recorded only where the effect is the method's *purpose*: [linear types](https://quality.stereobooster.com/linear-types.md) cover state (their data-race-freedom guarantee is that state discipline holding across threads — won by *forbidding* aliasing, not by reasoning about interleavings, so it's state, not bounded); [effect systems](https://quality.stereobooster.com/effect-systems.md) cover state and local I/O. - *Measurement* — measures a quantity rather than verifying a property: [profiling](https://quality.stereobooster.com/profiling.md), [microbenchmarking](https://quality.stereobooster.com/microbenchmarking.md), [load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md). What separates these is the *amount* of code measured — function, process, system — which this axis does not carry. - *Live observation* — takes the effect as given and watches: [monitoring](https://quality.stereobooster.com/monitoring-and-observability.md), [contracts and runtime assertions](https://quality.stereobooster.com/contracts-and-runtime-assertions.md), and [parallel run](https://quality.stereobooster.com/parallel-run.md) are the *observe* levels of the [same check at three rigor levels](https://quality.stereobooster.com/axes.md#the-same-check-at-three-rigor-levels) — a metric, a predicate, a difference against a reference. Their exposure to the live system is the [input axis](https://quality.stereobooster.com/input.md); their cost is the Heuristic/Empirical guarantee; scope doesn't apply. (Contrast [chaos engineering](https://quality.stereobooster.com/chaos-engineering.md): it *injects* the faults, controlling the space, so it earns a scope.) - **Reproducibility caps the scope at bounded; only aggregation reaches unbounded.** A method whose verdict needs a run it can reproduce — replay it ([deterministic simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md)), shrink to it ([property-based testing](https://quality.stereobooster.com/property-based-testing.md)), re-run both sides ([differential testing](https://quality.stereobooster.com/differential-testing.md)) — handles the network's *closed* nondeterminism but stops at the open distribution. Only statistical aggregation crosses it: [statistical testing](https://quality.stereobooster.com/statistical-testing.md) judges a sampling system (a real LLM) by a distribution over *n* runs, where no single run is a verdict and none needs reproducing. - **Effect scope is not [guarantee](https://quality.stereobooster.com/guarantee.md).** The two are independent: [deterministic simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md) and [theorem proving](https://quality.stereobooster.com/theorem-proving.md) both reach concurrent distributed code, yet one promises only *the cases tried passed* and the other a proof. And a guarantee can be inflated by *reducing* the effect: example tests on a distributed system earn *the cases tried passed*, but only over the enumerable slice they pretend the system is — worth less than a weaker claim against the real thing. Verification runs out as nondeterminism rises. The chart plots each method's guarantee against its nondeterminism scope — the one ordered effect dimension, I/O and state left off. Strong guarantees cluster on the left, in the closed-space effects owned by the formal and type families that *prove* or *exhaust*; the top-right stays empty, because at **unbounded** nondeterminism nothing beats *Empirical*. ## Where it comes from: Koka effects and Google size
![Effect lattice: total at the bottom, io at the top.](https://quality.stereobooster.com/assets/effect-lattice-light.svg#only-light)
These three dimensions are a recomposition of a richer vocabulary. The fullest is [Koka's](https://quality.stereobooster.com/effect-systems.md) effect system (Leijen 2014)[^leijen2014] (rooted in the original effect calculus (Lucassen and Gifford 1988)[^lucassen1988]): it names everything a function *does* beyond returning a value — throw, loop forever, read mutable state, perform I/O — tracking each as a *row* the type system carries next to the return type. | effect | the code may… | | ------ | ------------- | | `total` | nothing — terminate with a value | | `exn` | raise an exception | | `div` | not terminate (diverge) | | `pure` | throw or loop, but no observable side effect | | `ndet` | return different results for the same input | | `read⟨h⟩` `write⟨h⟩` `alloc⟨h⟩` | touch the heap `h` | | `st⟨h⟩` | hold mutable state in heap `h` | | `console` `net` `fsys` `ui` | reach the terminal, network, filesystem, or UI | | `io` | do real input/output | Effects don't fall on a single scale from "less" to "more." One effect sits below another only when its row is contained in the other's, so many pairs are incomparable: code that may loop (`div`) and code that holds state (`st⟨h⟩`) ask for different things from a test — a termination check vs state control. Only `total` (nothing to control) and `io` (everything) are common to all comparisons, as the lattice's bottom and top. Our testing cut keeps what changes the method and drops what doesn't: - `exn` and `div` fall away — a thrown exception or a timeout changes the assertion, not the technique. - `read` / `write` / `alloc` collapse to **state** — setup, teardown, order-dependence. - `console` / `net` / `fsys` / `ui` collapse around **network / distribution** — `net` means another machine and partial failure (a large test); local I/O is mild. - the **magnitude of nondeterminism** is an addition Koka doesn't carry at all. Read from the outside, the same effect is what *Software Engineering at Google* (Winters et al. 2020)[^winters2020] calls a test's **size** — the resources a run may use, defined by exactly what the code is allowed to touch: | Koka effect of the code | smallest Google test **size** it permits | this axis's tags | | ----------------------- | ---------------------------------------- | ---------------- | | `pure`, `st⟨h⟩` | **small** — single process, single thread, no I/O | `none`, `state` | | `io` without `net`, `ndet` | **medium** — one machine; threads + local I/O | `local I/O`, `bounded nondeterminism` | | `io` | **large** — multiple machines, real network | `network`, `bounded nondeterminism` | **Small** is deterministic by construction — single thread, no I/O — so any nondeterminism in it is *incidental* (a clock read, a seeded RNG, randomized iteration order over a Go map or a Python set) and is frozen, seeded, or sorted away before the ladder's tiers apply. Genuine nondeterminism begins at **medium**, where threads and local I/O bring it in. And **unbounded** nondeterminism appears nowhere: the table tops out at *bounded*. Every source Koka and Google name is a closed, modelable space, a distributed system's losses and reorderings included, just spread over more machines. That is why the ladder carries a tier neither the lattice nor the size tiers do. (**State**, likewise, doesn't track size: in-process state sits at *small* and rides along at every tier.) Google's other notion, **scope** (one function ↔ the whole system), is the *amount* of code rather than what it touches. ## Reading it off the type When the language carries effects in the type — [Koka](https://koka-lang.github.io/koka/doc/index.html), [F\*](https://www.fstar-lang.org/), [Multicore OCaml](https://ocaml.org/manual/effects.html) — the effect is in the function's signature: a function typed `io` can't be tested as if it were `pure`, and the compiler enforces that before the test is written. ## Referenced by - [Oracle](https://quality.stereobooster.com/oracle.md) · The axes - [The axes](https://quality.stereobooster.com/axes.md) · The axes - [Effect systems](https://quality.stereobooster.com/effect-systems.md) · Methods - [Statistical and sampling testing](https://quality.stereobooster.com/statistical-testing.md) · Methods - [Choosing methods](https://quality.stereobooster.com/choosing.md) · Overview - [Glossary](https://quality.stereobooster.com/glossary.md) · Overview ## References [^stockfish2026]: The Stockfish Project. 2026. *[Statistical Methods and Algorithms in Fishtest](https://official-stockfish.github.io/docs/fishtest-wiki/Fishtest-Mathematics.html)*. . [^leijen2014]: Leijen, Daan. 2014. "[Koka: Programming with Row Polymorphic Effect Types](https://arxiv.org/pdf/1406.2061)." *Proceedings of MSFP 2014 (Mathematically Structured Functional Programming)* 153: 100–126. . [^lucassen1988]: Lucassen, John M., and David K. Gifford. 1988. "[Polymorphic Effect Systems](https://ailang.sunholo.com/assets/files/lucassen-gifford-1988-0ae4870d6303ed164b78696901c7f22d.pdf)." *Proceedings of the 15th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL '88)*, 47–57. . [^winters2020]: Winters, Titus, Tom Manshreck, and Hyrum Wright, eds. 2020. *[Software Engineering at Google: Lessons Learned from Programming Over Time](https://abseil.io/resources/swe-book/html/ch11.html)*. O'Reilly Media. . ## Acronyms - DST — deterministic simulation testing - RNG — random number generator