The oracle is how a method decides an output is correct. There are three branches — property (a machine check on one output) and judgment (a human or AI decides one output), both single-run; and probabilistic (a claim about the distribution of many runs), which sits over either of the first two or stands alone.

• property
  • equality
  • predicate
  • relation
  • errors
• judgment
  • human
  • AI
• probabilistic
  • aggregate
  • statistical
  • exact

Property — a checkable claim

A property is a claim the output must satisfy, computed by a fixed procedure. The kinds line up by the form of the check, written as a formula over the program $f$:

kind	the check	examples
equality	$f(x) = y$ ($y$ given)	snapshot
	$f(x) = f'(x)$ ($y$ from a reference $f'$)	differential
predicate	$P(f(x))$	invariant $f(x) \geq 0$; type/schema $f(x) : T$; formal $f(x) \models \varphi$; threshold $m(f(x)) < c$; membership (supply-chain: dep $\notin$ advisory DB)
relation	$R(f(x), f(x'))$, $x' = t(x)$	metamorphic $f(\text{shuffle } x) = f(x)$, $f(2x) = 2 \cdot f(x)$
errors	$f(x)\downarrow$ (it just runs)	crash, hang, resource exhaustion, sanitizer trip

The kinds partition by the form written (a value, a unary predicate, a relation, or nothing), even though they nest logically by strength ($= y$ is a special predicate; a relation is a predicate over two outputs; $\downarrow$ is the weakest).

equality splits by the provenance of $y$: written out by hand → authored (assert f(2) == 4, a decision-table row); recorded → snapshot / approval (catches change, not correctness); a reference computes it → differential (relative correctness, only as good as the reference $f'$). An example test is oracle-agnostic rather than equality, because its author picks the form of the assertion and not only the expected value.

predicate spans a partial constraint many outputs pass — an ad-hoc invariant (one holding across all inputs, $\forall x.\, P(f(x))$), a type / schema, a machine-checked formal spec, a deterministic threshold ($m(f(x)) < c$ on a single output, e.g. a clone-similarity score), or a membership test (supply-chain hygiene: "this dependency is not in the advisory database"). A threshold over a distribution of runs is a different kind — probabilistic.

errors is the one kind with no authored claim — the runtime or a sanitizer (ASan / UBSan / MSan / TSan) supplies the check: an unexpected abnormal termination (a crash, hang, or uncaught exception), where memory-unsafety / UB is a bug by definition. An exception the contract documents (a named ValueError, not a bare crash) is the program working as specified; fuzzing harnesses catch those and treat only the rest as findings. It's still "computed by a fixed procedure": the procedure is just the runtime.

Judgment — a human or AI decides

When no checkable claim is writable — is this answer helpful? is this translation good? is this UI right? — a judge forms the verdict, with no fixed procedure.

• human — a person judges the run's output (raters, manual QA, exploratory testing). Judging the code rather than its output is code review — a process method, off this axis.
• AI — a model judges: LLM as judge, reward models. Historically the human oracle was the manual, last-resort option; AI makes the judgment branch automatable at scale (the case where AI is the oracle). Caveat: the judge is itself fallible, biased, and non-deterministic, so the judge itself must be validated, and its verdict is a distribution, not a single answer.

Probabilistic — a claim about the distribution

When the SUT or the judge is non-deterministic — LLMs, flaky concurrency, statistical code, the strength of a chess engine — no single run is a verdict. The oracle is a claim about a parameter of the output distribution — $\theta(D_f(x)) \bowtie c$, a parameter $\theta$ of the distribution $D_f(x)$ against a bound $c$: a win-rate, a pass-rate, a tail latency, an error budget, an A/B lift. The oracle obtains that parameter and compares it to the bound. Dobslaw et al. call this treating correctness as "a distribution of outcomes rather than a binary property" (Dobslaw et al. 2025)¹. This is the oracle of statistical / sampling testing; the sub-kinds differ in how the parameter is obtained, and how much the decision promises:

• Aggregate. The oracle summarizes a fixed number of runs — majority vote, mean, pass-rate $\geq$ a bound — and compares the summary to a threshold chosen by judgment. There's no controlled error rate: a "95% over 100 runs" gate bounds neither the odds of waving through a bad build nor the odds of rejecting a good one. This is the hope-for-the-best end, common in LLM evals and SLO checks; it caps at an empirical guarantee, and earns only heuristic where the threshold itself is a judgment rather than a measured bound.
• Statistical. The oracle states a hypothesis and controls the false-accept / false-reject rates, most often over two builds' metric distributions — a latency, a pass-rate. This is the statistical-confidence end. A fixed-sample test sets the number of runs in advance and a sequential test stops as soon as the evidence is decisive; the split is a sample-size choice inside the kind, and both control error where aggregate does not.
• Exact. No sampling at all: the parameter is computed over the whole state space, and the property checked against its true value. Exact probabilistic model checking (PRISM, Storm) solves a PCTL / CSL probability over every reachable state, so the verdict is sound within the model's bounds rather than a confidence interval. It is the only probabilistic sub-kind that reaches an exhaustive guarantee, and only because its input enumerates the space instead of sampling it.

A probabilistic oracle judges no individual run — a single lost game isn't a bug. Usually it sits on top of a per-run signal, a predicate's pass/fail or a human or AI judge's rating, and turns the sample into a verdict (a pass-rate is a predicate sampled; inter-rater agreement is judges sampled). But it needn't: a win-rate or a latency distribution has no per-run verdict underneath at all. Caveat: the test assumes roughly i.i.d. trials and a well-defined per-trial signal, so for truly open-ended output the aggregate is only as meaningful as that signal.

It's the only oracle available at the unbounded tier of the nondeterminism ladder — variation that can be neither removed nor modeled, only sampled — and it also judges noisy-but-bounded measurements like latency.

Correspondence to the literature

The standard oracle survey (Barr et al. 2015)² classifies oracles by their source; these kinds map onto it:

• provenance (authored / recorded / reference) ≈ Barr's specified vs derived — for us a sub-split of equality, not the spine.
• errors ≈ Barr's implicit.
• judgment / human ≈ Barr's human; we split out AI (a judge type that didn't exist in 2015).
• metamorphic is defined as a relation over two or more runs ($n \geq 2$) (Chen et al. 2018)³ — our relation kind.
• probabilistic has no counterpart in the survey. Barr's probabilistic test oracle maps one run to the interval $[0,1]$ (Barr et al. 2015)² — a softened verdict on a single output, where ours is a claim about many. Dobslaw et al.'s aggregated oracle is the match (Dobslaw et al. 2025)¹.

Sorting by form of the check separates the three things Barr's single derived bucket conflates: differential (value from a reference) and recorded (value from a prior run) are two provenances of equality, and metamorphic is a relation.

Related

Same name, different thing — Invariant: contract-lifetime vs across-inputs

The word invariant names different things. In Design by Contract it's a predicate on an object's state that holds across its whole lifetime — before and after every public method. In property-based testing and on the oracle axis it's a predicate that holds across inputs ($\forall x.\ P(f(x))$ — sort(xs) comes out ordered), which in contract terms is a postcondition, not the lifetime invariant. (Hoare logic adds a third: the loop invariant, across iterations.) The glossary lays out all three.

Same name, different thing — Property: the PBT claim vs the whole oracle branch

Property-based testing historically calls the claim it checks a "property" — usually a single predicate or a relation. On this axis, property is broader: it names the whole computed branch (equality / predicate / relation / errors), as opposed to human or AI judgment. So the property-based-testing sense is just the predicate and relation kinds within the property branch.

Referenced by

• Effect scope · The axes
• Guarantee · The axes
• The axes · The axes
• Contracts as specifications · Methods
• Metamorphic testing · Methods
• Methods · Methods
• Snapshot and approval testing · Methods
• How AI fits into software quality · AI
• Glossary · Overview

References

---

1. Dobslaw, Felix, Robert Feldt, Juyeon Yoon, and Shin Yoo. 2025. Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy. arXiv:2503.00481. https://doi.org/10.48550/arXiv.2503.00481. ↩↩

2. Barr, Earl T., Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. "The Oracle Problem in Software Testing: A Survey." IEEE Transactions on Software Engineering 41 (5): 507–25. https://doi.org/10.1109/TSE.2014.2372785. ↩↩

3. Chen, Tsong Yueh, Fei-Ching Kuo, Huai Liu, et al. 2018. "Metamorphic Testing: A Review of Challenges and Opportunities." ACM Computing Surveys 51 (1): 1–27. https://doi.org/10.1145/3143561. ↩