Effect is what the code under test touches — state, I/O, and nondeterminism. It is a property of the code, not of a method, which sets it apart from input, oracle, and guarantee. Effect scope is the same cut read the other way round: the richest effect a method can verify faithfully.

What changes the method is a coarse cut of the code's effect — three dimensions:

• Nondeterminism
  • enumerable (by hand)
  • bounded
  • unbounded
• I/O
  • local
  • network (distributed)
• State

A code's effect is the set of tags it carries (for example, [state, local I/O, bounded ndet]). Only the dimensions that are genuinely in play get tagged; a pure function carries none, which is exactly why it's trivial to test. Each tag is one more thing a test has to handle. The magnitude of nondeterminism is the dominant dimension, so it gets its own tiers; I/O level and state are coarser.

The nondeterminism ladder

Incidental nondeterminism comes off first. When it isn't part of the behavior under test — a timestamp written into a template, a tie-break RNG — seeding or freezing the clock restores a deterministic test, and most flakiness lives here. When the choice does affect the result the nondeterminism is genuine, and the topmost tier that fits applies:

1. Enumerate it by hand — when the space is tiny. Some nondeterminism can't be removed without removing the thing under test: the interleavings of two promises or two threads are the behavior. Two or three orderings fit in as many explicit cases.
2. Exhaust or explore it — when it's bounded but beyond hand enumeration. An interleaving explorer like loom — systematic concurrency testing — runs every permitted schedule of the real program; model checking checks an abstract state machine (TLA+ / TLC); deterministic simulation testing seeds every source of nondeterminism so a fuzzer can drive a reproducible space (FoundationDB). (Model checking is exhaustive over a model; DST samples a controlled, replayable space — both reach orderings hand enumeration never would.)
3. Sample it — when it's unbounded. Some nondeterminism can be neither seeded nor modeled — there's no closed space to enumerate, only a distribution to sample: a real LLM's outputs, or a statistical quantity no single run can settle. (A network is not this case — its losses and reorderings are a closed space TLA+ verifies and DST replays; that's bounded, tier 2.) One run is no longer a verdict: run it n times and judge the distribution — a probabilistic oracle. Stockfish decides every patch by playing a batch of games against the current version rather than by any single game (The Stockfish Project 2026)¹; LLM eval suites that gate on a pass-rate do the same.

Unaccounted nondeterminism is exactly a flaky test — a live source nobody chose a tier for, so the same code passes or fails by luck of the schedule, the clock, or the seed.

Which method fits

Most methods handle the easy combinations; nondeterminism and distribution are what force a specialist:

Method	Scope
Example / property-based / fuzzing	any effect up to bounded nondeterminism — above it one run isn't a verdict, and fuzzing can't triage a crash it can't reproduce
Model checking	concurrency + state, abstracted — not faithful real-world io
Deterministic simulation testing	concurrent / distributed, sealed and reproducible
Monitoring	live io / distributed — needs a deployed system
Statistical / sampling testing	unbounded nondeterminism — no space to model, only a distribution to sample (real LLMs, statistical outcomes)

Effect scope: a method's effect ceiling

Each method carries an effect scope, and selection is mechanical: for code with effect X, the candidates are the methods whose scope covers X. A method whose scope falls short doesn't fail loudly; it quietly verifies a reduced version of the system.

Effect scope is breadth — which effects a method covers, not how deeply a run exercises them. Depth is reachability: how much of a method's space a run exercises. Chaos engineering covers a wider effect scope than deterministic simulation testing yet is the shallower method.

A method earns a scope tag only when it controls the effect space — constructs it, or reasons over all of it — rather than merely observing, measuring, or applying regardless.

• Effect scope doesn't apply to observers or the indifferent. Several method categories control no effect space:

  • Effect-agnostic — applies to any code regardless of what it touches: clone detection, a plain type system, code review. Scope is recorded only where the effect is the method's purpose: linear types cover state (their data-race-freedom guarantee is that state discipline holding across threads — won by forbidding aliasing, not by reasoning about interleavings, so it's state, not bounded); effect systems cover state and local I/O.
  • Measurement — measures a quantity rather than verifying a property: profiling, microbenchmarking, load and stress testing. What separates these is the amount of code measured — function, process, system — which this axis does not carry.
  • Live observation — takes the effect as given and watches: monitoring, contracts and runtime assertions, and parallel run are the observe levels of the same check at three rigor levels — a metric, a predicate, a difference against a reference. Their exposure to the live system is the input axis; their cost is the Heuristic/Empirical guarantee; scope doesn't apply. (Contrast chaos engineering: it injects the faults, controlling the space, so it earns a scope.)

• Reproducibility caps the scope at bounded; only aggregation reaches unbounded. A method whose verdict needs a run it can reproduce — replay it (deterministic simulation testing), shrink to it (property-based testing), re-run both sides (differential testing) — handles the network's closed nondeterminism but stops at the open distribution. Only statistical aggregation crosses it: statistical testing judges a sampling system (a real LLM) by a distribution over n runs, where no single run is a verdict and none needs reproducing.

• Effect scope is not guarantee. The two are independent: deterministic simulation testing and theorem proving both reach concurrent distributed code, yet one promises only the cases tried passed and the other a proof. And a guarantee can be inflated by reducing the effect: example tests on a distributed system earn the cases tried passed, but only over the enumerable slice they pretend the system is — worth less than a weaker claim against the real thing.

Verification runs out as nondeterminism rises. The chart plots each method's guarantee against its nondeterminism scope — the one ordered effect dimension, I/O and state left off. Strong guarantees cluster on the left, in the closed-space effects owned by the formal and type families that prove or exhaust; the top-right stays empty, because at unbounded nondeterminism nothing beats Empirical.

Where it comes from: Koka effects and Google size

These three dimensions are a recomposition of a richer vocabulary. The fullest is Koka's effect system (Leijen 2014)² (rooted in the original effect calculus (Lucassen and Gifford 1988)³): it names everything a function does beyond returning a value — throw, loop forever, read mutable state, perform I/O — tracking each as a row the type system carries next to the return type.

effect	the code may…
total	nothing — terminate with a value
exn	raise an exception
div	not terminate (diverge)
pure	throw or loop, but no observable side effect
ndet	return different results for the same input
read⟨h⟩ write⟨h⟩ alloc⟨h⟩	touch the heap h
st⟨h⟩	hold mutable state in heap h
console net fsys ui	reach the terminal, network, filesystem, or UI
io	do real input/output

Effects don't fall on a single scale from "less" to "more." One effect sits below another only when its row is contained in the other's, so many pairs are incomparable: code that may loop (div) and code that holds state (st⟨h⟩) ask for different things from a test — a termination check vs state control. Only total (nothing to control) and io (everything) are common to all comparisons, as the lattice's bottom and top.

Our testing cut keeps what changes the method and drops what doesn't:

• exn and div fall away — a thrown exception or a timeout changes the assertion, not the technique.
• read / write / alloc collapse to state — setup, teardown, order-dependence.
• console / net / fsys / ui collapse around network / distribution — net means another machine and partial failure (a large test); local I/O is mild.
• the magnitude of nondeterminism is an addition Koka doesn't carry at all.

Read from the outside, the same effect is what Software Engineering at Google (Winters et al. 2020)⁴ calls a test's size — the resources a run may use, defined by exactly what the code is allowed to touch:

Koka effect of the code	smallest Google test size it permits	this axis's tags
pure, st⟨h⟩	small — single process, single thread, no I/O	none, state
io without net, ndet	medium — one machine; threads + local I/O	local I/O, bounded nondeterminism
io	large — multiple machines, real network	network, bounded nondeterminism

Small is deterministic by construction — single thread, no I/O — so any nondeterminism in it is incidental (a clock read, a seeded RNG, randomized iteration order over a Go map or a Python set) and is frozen, seeded, or sorted away before the ladder's tiers apply. Genuine nondeterminism begins at medium, where threads and local I/O bring it in.

And unbounded nondeterminism appears nowhere: the table tops out at bounded. Every source Koka and Google name is a closed, modelable space, a distributed system's losses and reorderings included, just spread over more machines. That is why the ladder carries a tier neither the lattice nor the size tiers do. (State, likewise, doesn't track size: in-process state sits at small and rides along at every tier.) Google's other notion, scope (one function ↔ the whole system), is the amount of code rather than what it touches.

Reading it off the type

When the language carries effects in the type — Koka, F*, Multicore OCaml — the effect is in the function's signature: a function typed io can't be tested as if it were pure, and the compiler enforces that before the test is written.

Referenced by

• Oracle · The axes
• The axes · The axes
• Effect systems · Methods
• Statistical and sampling testing · Methods
• Choosing methods · Overview
• Glossary · Overview

References

---

1. The Stockfish Project. 2026. Statistical Methods and Algorithms in Fishtest. https://official-stockfish.github.io/docs/fishtest-wiki/Fishtest-Mathematics.html. ↩

2. Leijen, Daan. 2014. "Koka: Programming with Row Polymorphic Effect Types." Proceedings of MSFP 2014 (Mathematically Structured Functional Programming) 153: 100–126. https://doi.org/10.4204/EPTCS.153.8. ↩

3. Lucassen, John M., and David K. Gifford. 1988. "Polymorphic Effect Systems." Proceedings of the 15th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL '88), 47–57. https://doi.org/10.1145/73560.73564. ↩

4. Winters, Titus, Tom Manshreck, and Hyrum Wright, eds. 2020. Software Engineering at Google: Lessons Learned from Programming Over Time. O'Reilly Media. https://abseil.io/resources/swe-book/html/ch11.html. ↩