# Coverage

Coverage measures *which parts of the program a test suite executed*:
**reachability, not correctness**. What it marks is where the test **safety net**
has holes: code no test reaches is code you cannot change safely.

## What it catches: a one-directional signal

Coverage is sound in exactly one direction. An uncovered line is hard proof the
suite never executes it, so no test in the suite can have caught a bug in it:
a real fact, established cheaply, and the cheapest gap-finder there is. A covered
line is not the converse. It proves only that the line *ran*, not that any
assertion pinned its behavior (execution ≠ verification).

That asymmetry splits coverage into two uses:

- **As a gap-hunter** ("which code has no test at all?") it rides the sound
  direction. An uncovered line is a real finding to act on, whether a developer
  reviews it, a CI check flags it on the diff, or a coverage-guided fuzzer steers
  toward it.
- **As an estimate of how well-tested the code is** it rides the direction
  coverage was never sound in, since a covered line is only *ran*. Here it is
  poor, because execution is one link of several that fault detection needs; the
  evidence and the measures that do the job are in [measuring test-suite
  effectiveness](https://quality.stereobooster.com/measuring-test-effectiveness.md).

Reading the gap-hunter's number as an effectiveness score is the
misinterpretation behind the "100% means done" reflex. Climbing to a more
adequate criterion certifies that more of a decision's logic was *exercised*, not
that its behavior was checked, so it does not move the effectiveness question at
all. What a covered line does **not** establish:

- **That the touched code is correctly tested.** `parseDate("2026-01-01")` with
  no assertion covers the same lines as one that checks the parsed value.
- **That the *right* code is covered.** A suite at 95% that misses the
  payment-authorization path is worse than one at 80% with the critical paths
  exercised.
- **That the inputs resemble production.** Every line hit by only happy-path
  shapes is empirically weaker than the same lines hit by varied input.
- **Anything outside example testing's reach.** Race conditions, integration
  failures, distributed state, and security boundaries don't surface in a
  line-coverage number, because the example tests that produce the coverage
  aren't designed to find them.

## The adequacy ladder

"100% coverage" is not a precise claim. Tools count different things and call all
of it *coverage*. In increasing adequacy:

| Form | What it requires | Adequacy |
| --- | --- | --- |
| **Statement / line** | the line was reached | weakest, most common; `if (a && b)` counts as covered even if the `&&` short-circuit was never exercised |
| **Branch / decision** | each branch of each conditional taken both ways | strictly more adequate than line; reveals untested `else` branches a one-line `if` hides |
| **Condition** | each Boolean sub-expression evaluates both true and false | independent of branch coverage — a test can satisfy one without the other |
| **MC/DC** (modified condition / decision) | each sub-expression shown to *independently* flip the decision (Hayhurst et al. 2001)[^hayhurst2001] | more adequate than condition; the criterion avionics standards demand at their highest levels |
| **Multiple-condition** (MCC) | every combination of a decision's conditions, the full truth table | exhaustive over the decision's logic (Zhu et al. 1997)[^zhu1997] |
| **Path** | every path through the control flow | intractable on real code — exponential in the branches, unbounded with loops |

In practice the **default toolchain stops at branch.** The mass-market
open-source tools — [coverage.py](https://coverage.readthedocs.io/), [JaCoCo](https://www.jacoco.org/jacoco/),
[c8](https://github.com/bcoe/c8)/[Istanbul](https://istanbul.js.org/), [SimpleCov](https://github.com/simplecov-ruby/simplecov), `go test
-cover` — report line and branch and nothing above it; condition, MC/DC, and MCC
stay the preserve of newer opt-in flags and commercial suites. So for the
overwhelming majority of projects the most adequate criterion available is branch,
which never enumerates an input space and so stays a *signal*, not a proof. The
one corner where a coverage *criterion* stops being a reachability measure and
becomes a bounded proof — MC/DC or MCC over finite, deterministic logic with a
real oracle — is a method in its own right: [exhaustive
coverage](https://quality.stereobooster.com/exhaustive-coverage.md).

Where coverage stays a signal, the measurement itself is not canonical:

- "100% line coverage" on code with rich Boolean expressions is much weaker than
  "100% branch coverage" on the same code.
- Denominators differ. Some tools count executable lines, others count blank
  lines, comments, imports, and declarations, so an untested file of 200 `import`
  lines scores differently per tool.
- Two tools on the same code and suite can report numbers tens of points apart
  from definitional choices alone.

The discipline that follows: one tool per project, the coverage flavor written
down, and the *delta* trusted (this PR moved coverage by N points) over the
absolute (we have 84%). Cross-project or cross-language comparisons are largely
meaningless without normalizing the definition.

The logic criteria disagree over less than line and branch do. Line and branch
have no canonical denominator even in principle (the divergence is arbitrary
text-counting); the logic criteria disagree only over how to *parse the
decision*, a principled and bounded disagreement:

| Criterion | Denominator anchors to | What two tools can disagree on |
| --- | --- | --- |
| Condition | the decision's Boolean structure | what counts as a *condition* — usually the operands of short-circuit `&&` / `\|\|`, but tools differ on bitwise `&` / `\|`, ternaries, relational leaves, and whether a Boolean variable is traced back to its definition |
| MC/DC | the structure **plus** a chosen form of the criterion | all of the above, plus *unique-cause* vs *masking* MC/DC, which accept different suites on coupled conditions (Chilenski 2001)[^chilenski2001]; GCC's implementation, for one, is masking (Kvalsvik 2025)[^kvalsvik2025] |
| Multiple-condition (MCC) | the full truth table | only the structural choice above; once the `n` conditions of a decision are fixed, 2ⁿ is pure combinatorics |

## Where coverage helps

- **At diff time: "did this PR leave new code untested?"** The uncovered new
  lines in a PR are a real signal for the author and reviewer, surfaced rather
  than blocked on.
- **As a coverage-regression gate.** If a module was covered yesterday and not
  today, something changed: new code without tests, or a deleted test. A
  significant drop is worth investigating; a few points of drift is noise.
- **On critical-path code.** A payment processor, an auth-token validator, a
  parser of untrusted input: low coverage on these specific modules is itself a
  finding, and more reliable on a small, well-bounded module than on a
  codebase-wide aggregate.
- **As a minimum bar where a regime mandates it.** Where a standard fixes the
  structural-coverage criterion, coverage is a compliance artifact, not a quality
  proxy: avionics under DO-178C, and medical-device software under IEC 62304,
  which requires documented unit verification, more rigorous at higher safety
  classes but without DO-178C's specific MC/DC mandate (IEC 2006)[^iec62304].
- **As a steering signal for input generation.** Coverage-guided
  [fuzzers](https://quality.stereobooster.com/fuzzing.md) ([libFuzzer](https://llvm.org/docs/LibFuzzer.html),
  [AFL++](https://github.com/AFLplusplus/AFLplusplus), [Atheris](https://github.com/google/atheris)) use line/branch coverage as the
  fitness function driving input mutation toward unexplored paths; the same loop
  on a property-based generator is coverage-guided property-based testing
  (Lampropoulos et al. 2019)[^lampropoulos2019]. Here coverage is the instrument, not the score.

## Match the criterion to the region

Even with the tools to measure it, full coverage everywhere is rarely
cost-justified. For a single decision of `n` conditions, the criteria that reason
about its Boolean structure diverge sharply in cost:

| Criterion | Tests for a decision of `n` conditions | `n` = 3 | `n` = 10 |
| --- | --- | --- | --- |
| Condition | 2 | 2 | 2 |
| MC/DC | `n` + 1 | 4 | 11 |
| Multiple-condition (MCC) | 2ⁿ (full truth table) | 8 | 1024 |

Condition coverage costs two tests at any size — all conditions true, then all
false — which is also why it is the weakest of the three: it never has to show a
condition *changed* the decision. MC/DC forces at least `n` + 1, demonstrating
each condition's independent effect; MCC enumerates every combination, and MC/DC
is its tractable substitute. Marginal cost climbs steeply while the assurance
each level buys falls off, so a single uniform target across a whole system
spends the most effort exactly where it returns the least.

The economically coherent move is to tier the *criterion* to the region's blast
radius, not to pick one percentage. DO-178C already does this, scaling the
required structural-coverage criterion to the Design Assurance Level (MC/DC at
Level A (RTCA 2011a)[^do178c], decision coverage at Level B, statement at Level C), and the
logic generalizes past avionics: demand branch or MC/DC on the payment/auth/parser
core, accept line coverage on the periphery, and reserve MCC for the few decisions
where exhausting the truth table is both tractable and worth it.

## Where coverage doesn't help

Mandating a coverage target ("reach 80%") invites Goodhart's-law gaming with
assertion-light tests.

## Tools

| Ecosystem        | Tool                                                                    | Notes                                                                |
| ---------------- | ----------------------------------------------------------------------- | -------------------------------------------------------------------- |
| JavaScript / TS  | **[c8](https://github.com/bcoe/c8)** (V8 native), **[Istanbul](https://istanbul.js.org/) / nyc**, [Vitest](https://vitest.dev/)'s built-in coverage, **[Monocart](https://quality.stereobooster.com/monocart.md)** | c8 uses V8 source coverage, nyc instruments — different numbers; Monocart merges V8 coverage across runners (Vitest + [Playwright](https://quality.stereobooster.com/playwright.md)) |
| Python           | **[coverage.py](https://coverage.readthedocs.io/)**, **[pytest-cov](https://pytest-cov.readthedocs.io/)** (wrapper)                               | Branch coverage opt-in via `--branch`                                |
| Java             | **[JaCoCo](https://www.jacoco.org/jacoco/)**, Cobertura                                                   | JaCoCo is the de facto default; supports line, branch, and complexity |
| Ruby             | **[SimpleCov](https://github.com/simplecov-ruby/simplecov)**                                                           | Line by default; branch coverage as of Ruby 2.5+                     |
| Go               | **`go test -cover`** (built-in), **[gocov](https://github.com/axw/gocov)**, **[gcov2lcov](https://github.com/jandelgado/gcov2lcov)**               | Statement coverage; no branch coverage in the standard tool          |
| Rust             | **[cargo-llvm-cov](https://github.com/taiki-e/cargo-llvm-cov)**, **[grcov](https://github.com/mozilla/grcov)**, **[cargo-tarpaulin](https://github.com/xd009642/tarpaulin)**                      | [llvm-cov](https://llvm.org/docs/CommandGuide/llvm-cov.html) is the modern default; tarpaulin is older and Linux-only    |
| C / C++          | **[gcov](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html) + [lcov](https://github.com/linux-test-project/lcov)**, **llvm-cov**                                           | Line and branch; both added opt-in MC/DC in 2024 (GCC 14 `--conditions`, Clang 18 `-fcoverage-mcdc`)  |
| .NET             | **[Coverlet](https://github.com/coverlet-coverage/coverlet)**, dotCover                                                  | Coverlet is the open-source default; integrates with `dotnet test`   |
| Elixir           | **[ExCoveralls](https://github.com/parroty/excoveralls)**                                                         | Line coverage; integrates with Coveralls reporting                   |
| Logic (MC/DC, MCC) | **[Testwell CTC++](https://www.verifysoft.com/en_ctcpp.html)**                                       | Cross-language (C/C++/Java/C#); the rare tool measuring multiple-condition coverage and MC/DC. Commercial |
| Diff coverage on PRs | **[diff-cover](https://github.com/Bachmann1234/diff_cover)**, GitLab MR coverage (native)                       | Flag uncovered new lines from the coverage report + git diff in CI; no third-party service needed |

## Evidence

When Google surfaced both a project-level coverage number and patch-level
uncovered lines, authors acted on the patch-level signal while the project-level
number sat largely ignored (Ivankovic et al. 2019)[^ivankovic2019] — the diff-time use is the one the
empirical record supports.

## Classification

- **Quality dimensions:** Maintainability — measures the reachability a test suite achieves, not its correctness.
- **Area:** Any codebase with an automated test suite; diff-time gap-finding in CI.

## Referenced by

- [Dead-code detection](https://quality.stereobooster.com/dead-code-detection.md) · Methods
- [Exhaustive coverage (MC/DC, MCC)](https://quality.stereobooster.com/exhaustive-coverage.md) · Methods
- [Git-history hotspots](https://quality.stereobooster.com/git-hotspots.md) · Methods
- [Measuring test-suite effectiveness](https://quality.stereobooster.com/measuring-test-effectiveness.md) · Methods
- [Mutation testing](https://quality.stereobooster.com/mutation-testing.md) · Methods
- [Test suite minimization](https://quality.stereobooster.com/test-suite-minimization.md) · Methods
- [The test suite as an object](https://quality.stereobooster.com/test-suite.md) · Methods
- [Monocart](https://quality.stereobooster.com/monocart.md) · Recipes
- [TypeScript](https://quality.stereobooster.com/typescript.md) · Recipes
- [How AI fits into software quality](https://quality.stereobooster.com/ai.md) · AI
- [Conventional terminology](https://quality.stereobooster.com/terminology.md) · Conventional
- [Testing folklore](https://quality.stereobooster.com/testing-folklore.md) · Conventional
- [Choosing methods](https://quality.stereobooster.com/choosing.md) · Overview
- [Glossary](https://quality.stereobooster.com/glossary.md) · Overview

## References

[^hayhurst2001]: Hayhurst, Kelly J., Dan S. Veerhusen, John J. Chilenski, and Leanna K. Rierson. 2001. *[A Practical Tutorial on Modified Condition/Decision Coverage](https://ntrs.nasa.gov/citations/20010057789)*. NASA/TM-2001-210876. <https://ntrs.nasa.gov/citations/20010057789>.
[^zhu1997]: Zhu, Hong, Patrick A. V. Hall, and John H. R. May. 1997. "[Software Unit Test Coverage and Adequacy](https://www.cs.toronto.edu/~chechik/courses18/csc410/p366-zhu.pdf)." *ACM Computing Surveys* 29 (4): 366–427. <https://doi.org/10.1145/267580.267590>.
[^chilenski2001]: Chilenski, John Joseph. 2001. *[An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion](https://rosap.ntl.bts.gov/view/dot/42764/dot_42764_DS1.pdf)*. DOT/FAA/AR-01/18. U.S. Department of Transportation, Federal Aviation Administration, Office of Aviation Research. [https://rosap.ntl.bts.gov/view/dot/42764/dot\\\_42764\\\_DS1.pdf](https://rosap.ntl.bts.gov/view/dot/42764/dot\_42764\_DS1.pdf).
[^kvalsvik2025]: Kvalsvik, Jørgen. 2025. "[Modified Condition/Decision Coverage in the GNU Compiler Collection](https://arxiv.org/pdf/2501.02133)." *arXiv Preprint arXiv:2501.02133*, ahead of print. <https://doi.org/10.48550/arXiv.2501.02133>.
[^iec62304]: IEC. 2006. *[IEC 62304: Medical Device Software — Software Life Cycle Processes](https://webstore.iec.ch/en/publication/6792)*. <https://webstore.iec.ch/en/publication/6792>.
[^lampropoulos2019]: Lampropoulos, Leonidas, Michael Hicks, and Benjamin C. Pierce. 2019. "[Coverage Guided, Property Based Testing](https://lemonidas.github.io/pdf/FuzzChick.pdf)." *Proceedings of the ACM on Programming Languages* 3 (OOPSLA): 181:1–29. <https://doi.org/10.1145/3360607>.
[^do178c]: RTCA. 2011. *[DO-178C: Software Considerations in Airborne Systems and Equipment Certification](https://www.rtca.org/do-178/)*. <https://www.rtca.org/do-178/>.
[^ivankovic2019]: Ivankovic, Marko, Goran Petrović, René Just, and Gordon Fraser. 2019. "[Code Coverage at Google](https://storage.googleapis.com/gweb-research2023-media/pubtools/5172.pdf)." *Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '19)*, 955–63. <https://doi.org/10.1145/3338906.3340459>.

## Acronyms

- MC/DC — modified condition/decision coverage
- MCC — multiple-condition coverage