# Checklists

A checklist is a fixed list of items, verified in a fixed order at a fixed
point in a procedure, by the people carrying that procedure out. It targets one
failure: the step a competent operator knew about and skipped. US Federal
Aviation Regulation 121.315 states the target directly, requiring that a
cockpit check procedure be designed so a crew member need not rely on memory
for the items to be checked (Degani and Wiener 1993)[^degani1993].

The verification runs over the *procedure*, not the program. Nothing is
executed.

## What it catches

- **A known-required step, omitted.** The feature flag left on after the
  rollout, the debug logging that shipped, the credential not rotated at
  handover, the replication lag not checked before the failover.
- **A procedure that lives in one person's head.** Writing the list surfaces
  the steps only the engineer who built the system knows about.
- **Divergence between operators.** Two engineers running the same deployment
  in two different orders produce incidents that reproduce for one of them and
  not the other.
- **The step that pays off only when something else has already gone wrong.**
  Verifying that the backup restores, that the rollback path is exercised, that
  the on-call rotation has a name in it.

## What it does not catch

A checklist enumerates what someone already knew to write down, so it does
nothing about a failure mode nobody has met. Each item is also only as good as
the answer it gets: an operator who reads "backups verified" and answers yes
without looking has completed the list and checked nothing.

## Design decides whether it works

Degani and Wiener's field study of flight-deck checklists — crews observed in
operation, interviews at seven US airlines, three accident databases —
produced design rules (Degani and Wiener 1993)[^degani1993]:

- **State the value, not "done".** A response should name the status the item
  should be in — `replicas: 3`, `feature flag: off` — rather than "checked". A
  list answered with "done" verifies that the list was read.
- **Critical items first.** Items whose omission alone can cause the failure
  belong at the top, where an interruption is least likely to have happened
  yet, and that placement takes precedence over grouping items by system or by
  workflow. Pilots call them *killer items*.
- **Chunk a long list.** Subdivide it by system or by phase, into groups small
  enough to hold in memory. Position in the list is the only marker of how far
  the operator has got.
- **Verify after doing, not instead of doing.** Under *challenge-response* the
  operator configures the system from memory and then reads the list to confirm
  each item. Under *read-do* the list is the procedure itself, and once the
  sequence is interrupted a skipped item passes unnoticed.
- **Expect production pressure to degrade use.** The observed failure was crews
  calling several items as one group and answering in a batch, restoring the
  dependence on memory the list exists to remove.

## Automate every item a machine can decide

An item a program can decide does not belong on a human's list. "No
`console.log` left in the diff" is a [linter](https://quality.stereobooster.com/linters.md)
rule, "the migration is reversible" a CI job, "the ticket has an owner" a
required template field. Every item moved off the list is one the operator can
no longer forget.

Automation cannot absorb items needing judgment (is the rollback plan
executable, or only written down?), items needing action outside the machine
(support has been told, the maintenance window is announced), or items where
the measurement is automatic but the decision is not (the dashboard shows a 30%
traffic drop — is that expected at this hour?).

CIS Benchmarks and DISA STIGs began as documents an administrator worked
through by hand; most of their items are now machine-checkable content that
[OpenSCAP](https://www.open-scap.org/) or [kube-bench](https://github.com/aquasecurity/kube-bench) evaluates against a
host or a cluster.

## Tools: the checklists already in use

| Checklist | Point of use | What it holds |
| --- | --- | --- |
| [OWASP ASVS](https://owasp.org/www-project-application-security-verification-standard/) | Security requirement review | Numbered verification requirements at three levels of rigor, written to be cited individually in a contract or a test plan |
| [SRE launch coordination checklist](https://sre.google/sre-book/reliable-product-launches/) | Before a service takes production traffic | Google's questions on architecture and dependencies, capacity planning, failure modes, and rollout planning |
| [STRIDE](https://quality.stereobooster.com/threat-modeling.md) | Design review | Six threat categories walked per component |
| [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) | Agent and LLM feature review | Ten named risk categories; a coverage prompt, not a measurement |
| [Requirements quality characteristics](https://quality.stereobooster.com/requirements-quality-checking.md) | Specification draft | Ambiguity, inconsistency, incompleteness, unmeasurable requirements — the rule set a requirements engineer works a draft against |
| Pull-request template, definition of done | Every change | Team-authored; whatever automation has not yet absorbed |
| Incident runbook | During an outage | Ordered steps for a diagnosis or a recovery |

## When to use, when not

**Use when** the procedure repeats, its steps are known, omission is costly,
and the people running it work under interruption — deployment, failover,
release, incident response, production-readiness review.

**Do not use it as a defect-finding aid.** Handing a reviewer a checklist of
defect categories to read an artifact against is the use of checklists that has
been measured in software, and it did not help.

**Do not mandate it without implementing it.** A published list nobody was
trained on and nobody owns produces compliance and no change in outcome.

## Evidence

**No measured benefit in code inspection.** Hatton analyzed 308 inspections
carried out in workshops for industrial engineers and found no evidence that
checklists made a significant difference to faults found per inspection hour,
whether or not the inspector was experienced (Hatton 2008)[^hatton2008]. The same data
showed individual performance varying by a factor of 10, with individuals
finding about 53% of the faults and two-person teams 76% — who inspects
dominates whether they hold a list.

**Checklist reading is no better than ad hoc reading.** In a randomized
experiment, 48 graduate students in 16 three-person teams inspected two
requirements specifications using ad hoc, checklist, or scenario-based methods.
Checklist reviewers were no more effective than ad hoc reviewers, while
scenario reviewers — each given a different systematic procedure aimed at a
specific fault class — had a higher fault detection rate (Porter et al. 1995)[^porter1995]. Student
subjects, so the ordering of the three methods is the finding, not the size of
the gap.

**Large effects on omission-heavy clinical procedures.** The WHO's 19-item
surgical safety checklist was introduced at eight hospitals in eight countries;
across 3733 patients before and 3955 after, the death rate fell from 1.5% to
0.8% and inpatient complications from 11.0% to 7.0% (Haynes et al. 2009)[^haynes2009].
Before-and-after within each site, not randomized. A five-item central-line
insertion checklist — hand washing, full-barrier precautions, chlorhexidine,
avoiding the femoral site, removing unnecessary catheters — across 103 Michigan
ICUs and 375,757 catheter-days cut the median catheter-related bloodstream
infection rate from 2.7 to 0 per 1000 catheter-days at three months, with the
mean still down from 7.7 to 1.4 at 16 to 18 months (Pronovost et al. 2006)[^pronovost2006].

**Mandating the list is not implementing it.** Ontario made surgical safety
checklists policy across its acute care hospitals. Comparing 109,341 procedures
before adoption with 106,370 after at 101 hospitals, adjusted mortality moved
from 0.71% to 0.65% (odds ratio 0.91, P=0.13) and adjusted complications from
3.86% to 3.82% (P=0.29) — neither significant (Urbach et al. 2014)[^urbach2014]. The hospitals in
the WHO study joined a program and collected outcome data prospectively;
Ontario's received a policy.

**The transfer to software is unmeasured.** The clinical results are evidence
for checklists over procedures whose steps are physical, enumerable, and
omission-dominated, which is the shape a deployment or a failover also has.
Whether the effect carries over has not been measured. The software studies
tested a different use — reading code or a specification to find unknown
defects — and found nothing.

## Classification

- **Quality dimensions:** Reliability, Security, Functionality.
- **Area:** Repeated operational procedures with known steps and costly omissions — deployment, failover, release, incident response, production-readiness review, security configuration.

## Referenced by

- [How AI fits into software quality](https://quality.stereobooster.com/ai.md) · AI

## References

[^degani1993]: Degani, Asaf, and Earl L. Wiener. 1993. "[Cockpit Checklists: Concepts, Design, and Use](https://doi.org/10.1177/001872089303500209)." *Human Factors: The Journal of the Human Factors and Ergonomics Society* 35 (2): 345–59. <https://doi.org/10.1177/001872089303500209>.
[^hatton2008]: Hatton, Les. 2008. "[Testing the Value of Checklists in Code Inspections](https://www.leshatton.org/Documents/checklists_in_inspections.pdf)." *IEEE Software* 25 (4): 82–88. <https://doi.org/10.1109/MS.2008.100>.
[^porter1995]: Porter, Adam A., Lawrence G. Votta, and Victor R. Basili. 1995. "[Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment](https://www.cs.umd.edu/~basili/publications/journals/J56.pdf)." *IEEE Transactions on Software Engineering* 21 (6): 563–75. <https://doi.org/10.1109/32.391380>.
[^haynes2009]: Haynes, Alex B., Thomas G. Weiser, William R. Berry, et al. 2009. "[A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population](https://dash.harvard.edu/bitstreams/c5287a52-75bc-4131-9262-4c598fcde8cc/download)." *New England Journal of Medicine* 360 (5): 491–99. <https://doi.org/10.1056/NEJMsa0810119>.
[^pronovost2006]: Pronovost, Peter, Dale Needham, Sean Berenholtz, et al. 2006. "[An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU](https://www.nejm.org/doi/10.1056/NEJMoa061115)." *New England Journal of Medicine* 355 (26): 2725–32. <https://doi.org/10.1056/NEJMoa061115>.
[^urbach2014]: Urbach, David R., Anand Govindarajan, Refik Saskin, Andrew S. Wilton, and Nancy N. Baxter. 2014. "[Introduction of Surgical Safety Checklists in Ontario, Canada](https://doi.org/10.1056/NEJMsa1308261)." *New England Journal of Medicine* 370 (11): 1029–38. <https://doi.org/10.1056/NEJMsa1308261>.

## Acronyms

- SRE — site reliability engineering
- STIG — Security Technical Implementation Guide (DISA)
- STRIDE — Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege