# Conventional terminology The testing literature has accreted decades of vocabulary — *unit test*, *integration test*, *regression*, *acceptance*, *smoke*, *mock*, *stub*, *golden master*. The terms are useful as shorthand but cluster meanings that often differ between authors. ## Conventional taxonomy → the axes The unit/integration/e2e hierarchy is the dominant mental model in software testing. The project's pages use its terms — but the labels package multiple axes into one, which is the source of most "what counts as a unit test?" debates. | Term | Input | Oracle | Guarantee | Ambiguity | | ------------------------ | --------------- | ------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Unit test** | Fixed | Any | Empirical | Packs *narrow in scope* with *isolated by mocks*. The two aren't the same; most "unit tests" should arguably be behavioral tests against a port (Cooper 2013)[^cooper2013]. | | **Integration test** | Fixed | Any | Empirical | "Integration of what?" In some shops it means "two modules"; in others "module + DB"; in others "module + a real third-party service." | | **End-to-end (E2E) test**| Fixed | Any | Empirical | Often used interchangeably with *system test*, *acceptance test*. Strictly: drives the whole deployed system from its outermost surface. | All three rows carry the same defaults (a fixed input, an oracle the author writes as code, an empirical guarantee) unless one is chosen otherwise. The levels differ only in **scope** (how much code a run exercises) and **size** (the resources and I/O reach it needs), properties of the code and the run that Google's taxonomy names directly and the method axes deliberately leave out. ### Scope and size (Google's taxonomy) Google splits the two properties the unit/integration/e2e labels conflate, and treats them as *orthogonal* (Winters et al. 2020)[^winters2020]: - **Scope** — how much code a run exercises: *narrow* (one class or function), *medium* (a few components together), *broad* (the whole system). - **Size** — the resources a run is allowed, which is what actually governs its speed and flakiness: *small* (one process, one thread, no I/O, no sleeps — fast and deterministic), *medium* (one machine: localhost, a local database, the filesystem), *large* (multiple machines, real network, external services). | Conventional term | Scope | Typical size | | ------------------ | ------ | ------------ | | **Unit** | narrow | small | | **Integration** | medium | medium | | **End-to-end** | broad | large | The size column is only a default. The two properties come apart: a test that touches the disk or the clock is medium in size however narrow its scope, and it is the size that makes it slow and flaky. Google therefore classifies by size first and targets mostly-small suites, whereas the pyramid assigns one label per level and so covers scope and size together. ### Visibility: black-box, gray-box, white-box **Visibility** is what a method can observe of the system *while it runs*: the bandwidth of the channel between the method and the running code. It is set by the instrumentation available, not by what the person driving the method knows about the implementation. The three labels mark points on that one gradient rather than three kinds of testing, and each widening of the channel makes the next family of methods possible: | Channel | The method observes | What it unlocks | | --- | --- | --- | | None | the output, and whether the process died | random generation, [approval tests](https://quality.stereobooster.com/snapshot-testing.md), synthetic probing | | Crash and exit status | that *something* failed, not where | crash triage, black-box [fuzzing](https://quality.stereobooster.com/fuzzing.md), [Jepsen](https://jepsen.io/)-style fault testing | | Coverage bits | which edges a run reached | coverage-guided (gray-box) fuzzing, [coverage](https://quality.stereobooster.com/coverage.md) measurement, [mutation testing](https://quality.stereobooster.com/mutation-testing.md) | | Path constraints | the symbolic condition under which a run took its path | [concolic execution](https://quality.stereobooster.com/symbolic-execution.md), branch-distance guidance in [search-based testing](https://quality.stereobooster.com/search-based-software-testing.md) | | Source and semantics | the program text and its meaning | [symbolic execution](https://quality.stereobooster.com/symbolic-execution.md), [abstract interpretation](https://quality.stereobooster.com/abstract-interpretation.md), [types](https://quality.stereobooster.com/types.md), proof | Visibility is a third property the pyramid labels conflate, independent of both scope and size. A unit test driving a public API with no instrumentation is black-box at narrow scope; an end-to-end run under a coverage agent, with a fault injected into one chosen internal call, is white-box at broad scope. The equation *end-to-end = black-box, unit = white-box* packs together things that come apart in practice. It is not one of the [axes](https://quality.stereobooster.com/axes.md), because it is a property of the *situation* rather than of the method: the same method sits at the same coordinates whether the source is available or not. Visibility decides which coordinates are *available*. A generative-feedback input needs a feedback channel, so against a genuine black box the input axis collapses to fixed or generative-random. Every criterion on the [adequacy ladder](https://quality.stereobooster.com/coverage.md) needs instrumentation, so a coverage number is undefined without one. And a proof needs the semantics, which is why [temporal-logic falsification](https://quality.stereobooster.com/temporal-logic-falsification.md) exists at all: the Simulink model is opaque, so what remains is search against a robustness score. More channel is not automatically more bugs found, and the fuzzing literature is weak ground for the comparison. A survey of 32 fuzzing papers found problems in the experimental evaluation of every one, and the authors' own experiments showed those problems can produce wrong or misleading assessments (Klees et al. 2018)[^klees2018]. What the channel buys is definitional rather than measured: a black-box fuzzer *cannot* run the feedback loop at all, because the signal it would steer by does not reach it. ## Synonyms and near-synonyms Words that *look* like distinctions but usually aren't, or that collapse together under different authors. - **Example test** ↔ **automanual test** (Wayne 2018)[^wayne2018a]. The project uses *example-based* to emphasize the input axis; Wayne's *automanual* captures the same thing. - **Regression test** = *example test added after a bug fix*. The distinguishing feature is the *role* in the suite, not the *form*. Every regression test is an example test; not every example test is a regression test. - **Approval test** ↔ **snapshot test** ↔ **golden master test**. Same technique under three names. *Golden master* carries loaded vocabulary; the project standardizes on *approval* (or *snapshot* where the framework uses that term). - **Characterization test** (Feathers 2004)[^feathers2004] = an approval test applied to legacy code with no specification. The test pins down current behavior to make a refactor safe; correctness is a separate question. - **BDD** / **Gherkin** / **Specification by Example** — these are *styles* for writing example tests, not separate test families. Given/When/Then is a syntax; the test is still an example test underneath. - **Mock** ↔ **stub** ↔ **fake** ↔ **spy** (Fowler 2007)[^fowler2007]. *Mock* is routinely used for every kind of test double; Fowler separates them: - *Stub* — returns canned answers; tests against state. - *Mock* — verifies interactions; tests against calls. - *Fake* — a working implementation, just not the production one (in-memory DB, fake clock). - *Spy* — records calls for later assertion. - **Classicist** vs **mockist** TDD — Fowler's terms for the two schools. Classicist: test against state with real collaborators or fakes. Mockist: test against interactions with mocked collaborators. ## Referenced by - [The axes](https://quality.stereobooster.com/axes.md) · The axes - [Conventional](https://quality.stereobooster.com/conventional.md) · Conventional - [Glossary](https://quality.stereobooster.com/glossary.md) · Overview ## References [^cooper2013]: Cooper, Ian. 2013. *[TDD: Where Did It All Go Wrong?](https://www.youtube.com/watch?v=EZ05e7EMOLM)* . [^winters2020]: Winters, Titus, Tom Manshreck, and Hyrum Wright, eds. 2020. *[Software Engineering at Google: Lessons Learned from Programming Over Time](https://abseil.io/resources/swe-book/html/ch11.html)*. O'Reilly Media. . [^klees2018]: Klees, George, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. "[Evaluating Fuzz Testing](https://dl.acm.org/doi/pdf/10.1145/3243734.3243804)." *Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS '18)*, 2123–38. . [^wayne2018a]: Wayne, Hillel. 2018. *[Just a Whole Bunch of Different Tests](https://www.hillelwayne.com/post/a-bunch-of-tests/)*. . [^feathers2004]: Feathers, Michael. 2004. *Working Effectively with Legacy Code*. Prentice Hall PTR. [^fowler2007]: Fowler, Martin. 2007. *[Mocks Aren't Stubs](https://martinfowler.com/articles/mocksArentStubs.html)*. . ## Acronyms - BDD — behavior-driven development - CCS — Calculus of Communicating Systems - E2E — end-to-end - TDD — test-driven development