Skip to content

Software Quality

Testing folklore

Five popular testing prescriptions are folklore, repeated as settled practice without measured support. They are the pyramid, the 80% coverage target, TDD as universal design discipline, DRY for tests, and "100% coverage subsumes types". Each is a single-axis prescription offered as a complete answer to a multi-axis question.

The pyramid is the canonical example. The critique targets the doctrine; the underlying technique — example tests, fast feedback — usually survives it.

The pyramid

The test pyramid is the dominant mental model in software testing. It prescribes a suite shaped like a pyramid: many unit tests at the base, fewer integration tests in the middle, fewer still end-to-end tests at the top. The pyramid was first published in Succeeding with Agile (Cohn 2009)1. Martin Fowler's bliki canonized it (Fowler 2012)2. Almost every testing handbook published since assumes it.

The strongest version of the argument

Cohn's argument is economic, and it is reasonable (Cohn 2009)1.

Lower-level tests localize failure. A failing unit test identifies the class and method involved — the signal is "there is a bug and it is on line 47." Higher-level tests do not localize: a failing end-to-end test shows that something across many components is wrong, but finding what requires debugging.

Lower-level tests are fast and cheap. They run in milliseconds, in parallel, against fixtures, returning results before the change has left working memory.

Higher-level tests are slow, flaky, and expensive. They run through real I/O. They depend on environment. They break for reasons that have nothing to do with the change you made.

From these three premises, the pyramid follows:

  • More cheap tests, fewer expensive ones.
  • Failure localized as close to its cause as the tests allow.
  • Use expensive tests only for cross-component flows the cheap ones cannot cover.

The pyramid was a defensible default on 1990s build infrastructure.

What the pyramid gets wrong

One: it isn't supported by evidence

After substantial searching across academic literature, industrial postmortems, and engineering blogs, no peer-reviewed study or quantitative postmortem demonstrates that teams following a pyramid-shaped suite ship higher quality or faster software than teams that don't. The pyramid is a heuristic, not a finding.

The frequently-cited "80% faster feedback, 70% lower maintenance cost, 90% fewer production defects" numbers trace only to vendor blogs without primary sources; they are not measurements.

This does not mean the pyramid is wrong. It means it is folklore.

Two: it optimizes the wrong axis

The pyramid ranks tests by scope of test infrastructure — how much real system is brought up. That maps to scope and size (Google's terms), but only roughly: the pyramid mixes infrastructure scope with the shape of the code being tested. Either way it is a property of the code and the run, not of the method (see The Axes).

The pyramid has nothing to say about input or the oracle: whether the test is example-based or generative, whether its inputs come from property-based generation or fuzzing, whether its verdict comes from a metamorphic relation or a differential comparison against a reference. Two tests at the same place in the pyramid can reduce wildly different amounts of uncertainty depending on how their inputs are chosen and how their outputs are judged.

Nor about guarantee: a property test makes a claim about all inputs in a class; an example test makes a claim about one input. They are not equivalent at any level of the pyramid.

By ranking on infrastructure scope alone, the pyramid encourages teams to add example-based tests on pure or near-pure code instead of generative, differential, or metamorphic tests on the same code. Those techniques are exactly what the empirical record favors (Csmith, SQLancer, Volvo AUTOSAR, OSS-Fuzz). The pyramid does not tell you to avoid them; it just does not see them.

Three: it conflates two different ideas

"Unit test" packs two unrelated ideas into one label: small in scope and isolated from collaborators. They are not the same.

A test of a single function with no dependencies is both small and isolated by construction — there is nothing to isolate from. A test of a single function with mocked collaborators is small but artificially isolated. A test of a single collaborating cluster is small in conceptual scope but not isolated from internals. The "unit test" label flattens all three.

The mockist/classicist debate is the high-profile symptom of this collapse. Fowler's Mocks Aren't Stubs (Fowler 2007)3 maps the territory: mockist TDD couples tests to implementation (calls to collaborators); classic TDD couples tests to state. Both are legitimate; they trade off different things. The pyramid takes no position because it cannot see the distinction — both are "unit tests" to it.

TDD, Where Did It All Go Wrong (Cooper 2013)4 (the talk is video-only; the account here is drawn from multiple consistent written reviews) makes the further claim that most "unit tests" should be tests against the use-case port (the behavioral boundary of a feature), not against a class. By that view, the pyramid's "unit" layer is mostly miscategorized: what teams write are tiny behavioral tests, and the real integration and system distinctions cleave at different lines than the pyramid draws. The hierarchy is internally incoherent.

Cohn's strong arguments, in modified form

The pyramid's two strong arguments — localization and cost — survive in modified form:

Localization. Property-based tests localize as well as example tests, sometimes better. A shrinking property test that fails identifies a minimal counterexample. A failing example test identifies a known case. Both point to the function under test.

Cost. Tests with real I/O are slower than tests with mocks. But the relevant cost is cost per bug prevented, not cost per test run. A fuzzing campaign that costs CI minutes but finds a class of bugs no example test would catch may be cheaper than hundreds of example tests that cover only the cases the author imagined.

The pyramid solved the cost question by saying write the cheaper test. The right answer is write the test that finds more bugs per dollar. Those are not the same answer.

Other folklore in the same shape

"80% coverage" and other fixed thresholds

Pick a number, demand a suite meet it, treat the number as the quality signal. Once suite size is controlled for, coverage is only weakly to moderately correlated with fault-detection effectiveness (see coverage), and mutation testing is the sharper proxy for assertion strength — though it, too, is confounded by suite size (Papadakis et al. 2018)5.

The deeper failure mode is the same as the pyramid's: an input signal masquerading as a guarantee claim. High coverage from many example tests with weak assertions is still empirical-tier guarantee on the cases the author wrote down. The number doesn't change what the tests can prove.

Coverage is operationally useful at diff time — as a "did this patch leave new code unexercised?" signal — not as an org-wide KPI (Ivankovic et al. 2019)6.

"TDD as universal design discipline"

The original Beck framing of TDD is modest: write tests first as a habit; let the act of writing them shape the code's shape. Strong/maximal TDD inflates this into "TDD is the design discipline; testability is the design metric." That cost has a name — test-induced design damage (Hansson 2014)7: the maxim "Code that's hard to test in isolation is poorly designed", taken to its conclusion, produces architectures whose only purpose is making testing easier.

The empirical record on TDD is mixed: the better-controlled studies find a small gain in external quality bought at a cost to productivity, and a methodological autopsy documents why the literature keeps producing contradictory findings (Ghafari et al. 2020)8.

As folklore, "always TDD" picks one axis (process discipline) and declares it the answer. The other two axes — what bugs you catch and how strongly you catch them — are silent. Property tests, contracts, types, and reference oracles do work TDD does not, and vice versa.

"DRY for tests"

Refactor tests into shared helpers, mother objects, fluent builders, test-data factories — to remove duplication. The cost, when this goes wrong, is the same as DRY in production code: the abstraction couples tests to internals that should have been free to change. Brittle test suites whose maintenance cost overwhelms their value are this folklore's signature failure.

The defensible version is "avoid duplication of test data construction"; the folklore version is "unify every assertion shape". The first is hygiene; the second is test-induced design damage on the test suite itself.

"100% coverage subsumes types"

Robert C. Martin: "You don't need static type checking if you have 100% unit test coverage" (Martin 2016)9. The single-axis claim: coverage is the verification quantity; types are a separate axis made redundant by maximizing the first.

Falsified by direct measurement: a study that checked exactly this surface found ~15% of public JavaScript bug fixes in their sample would have been caught by gradual typing (Gao et al. 2017)10 — a conservative floor, since public bugs are typically already tested and reviewed. Three concrete code examples of bug classes that unit tests structurally miss — caught instead by types, property-based testing, and static analysis — are given in (Wayne 2017)11.

The deeper folk pattern: treating verification methods as substitutable (more of one displaces the need for another) rather than as complementary (each rules out a different bug class).

What folklore has in common

All five — the pyramid, fixed coverage thresholds, TDD as universal design discipline, DRY for tests, "100% coverage subsumes types" — share a shape:

  1. Pick one property (infrastructure scope; line coverage; process discipline; test reuse; method substitutability).
  2. Treat it as the answer.
  3. Stay silent on the rest.
  4. Travel widely.

The corrective is the same in each case: what kind of uncertainty does this method reduce, and which other methods reduce the uncertainties this one doesn't? That question is what the axes framework makes answerable, and what the Methods catalog answers in detail.

Defaults that don't survive inspection are still useful as defaults. But when defaults are promoted to doctrine — "we follow the testing pyramid", "our coverage must be 80%" — folklore is what the team ships with.

Design folklore covers the design-side prescriptions in the same shape: small files / small functions, complexity metrics as quality targets, GoF design patterns as universal design vocabulary, and DRY taken as no-duplication.

Referenced by

References

  1. Cohn, Mike. 2009. Succeeding with Agile: Software Development Using Scrum. Addison-Wesley. ↩↩

  2. Fowler, Martin. 2012. Test Pyramid. https://martinfowler.com/bliki/TestPyramid.html. ↩

  3. Fowler, Martin. 2007. Mocks Aren't Stubs. https://martinfowler.com/articles/mocksArentStubs.html. ↩

  4. Cooper, Ian. 2013. TDD: Where Did It All Go Wrong? https://www.youtube.com/watch?v=EZ05e7EMOLM. ↩

  5. Papadakis, Mike, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. "Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults." Proceedings of the 40th International Conference on Software Engineering (ICSE '18), 537–48. https://doi.org/10.1145/3180155.3180183. ↩

  6. Ivankovic, Marko, Goran Petrović, René Just, and Gordon Fraser. 2019. "Code Coverage at Google." Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '19), 955–63. https://doi.org/10.1145/3338906.3340459. ↩

  7. Hansson, David Heinemeier. 2014. Test-Induced Design Damage. https://dhh.dk/2014/test-induced-design-damage.html. ↩

  8. Ghafari, Mohammad, Timm Gross, Davide Fucci, and Michael Felderer. 2020. "Why Research on Test-Driven Development Is Inconclusive?" Proceedings of ESEM '20, 1–10. https://doi.org/10.1145/3382494.3410687. ↩

  9. Martin, Robert C. 2016. Type Wars. The Clean Code Blog. https://blog.cleancoder.com/uncle-bob/2016/05/01/TypeWars.html. ↩

  10. Gao, Zheng, Christian Bird, and Earl T. Barr. 2017. "To Type or Not to Type: Quantifying Detectable Bugs in JavaScript." Proceedings of the 39th International Conference on Software Engineering (ICSE '17), 758–69. https://doi.org/10.1109/ICSE.2017.75. ↩

  11. Wayne, Hillel. 2017. Uncle Bob and Silver Bullets. https://www.hillelwayne.com/post/uncle-bob/. ↩