Skip to content

Software Quality

Checklists

A checklist is a fixed list of items, verified in a fixed order at a fixed point in a procedure, by the people carrying that procedure out. It targets one failure: the step a competent operator knew about and skipped. US Federal Aviation Regulation 121.315 states the target directly, requiring that a cockpit check procedure be designed so a crew member need not rely on memory for the items to be checked (Degani and Wiener 1993)1.

The verification runs over the procedure, not the program. Nothing is executed.

What it catches

  • A known-required step, omitted. The feature flag left on after the rollout, the debug logging that shipped, the credential not rotated at handover, the replication lag not checked before the failover.
  • A procedure that lives in one person's head. Writing the list surfaces the steps only the engineer who built the system knows about.
  • Divergence between operators. Two engineers running the same deployment in two different orders produce incidents that reproduce for one of them and not the other.
  • The step that pays off only when something else has already gone wrong. Verifying that the backup restores, that the rollback path is exercised, that the on-call rotation has a name in it.

What it does not catch

A checklist enumerates what someone already knew to write down, so it does nothing about a failure mode nobody has met. Each item is also only as good as the answer it gets: an operator who reads "backups verified" and answers yes without looking has completed the list and checked nothing.

Design decides whether it works

Degani and Wiener's field study of flight-deck checklists — crews observed in operation, interviews at seven US airlines, three accident databases — produced design rules (Degani and Wiener 1993)1:

  • State the value, not "done". A response should name the status the item should be in — replicas: 3, feature flag: off — rather than "checked". A list answered with "done" verifies that the list was read.
  • Critical items first. Items whose omission alone can cause the failure belong at the top, where an interruption is least likely to have happened yet, and that placement takes precedence over grouping items by system or by workflow. Pilots call them killer items.
  • Chunk a long list. Subdivide it by system or by phase, into groups small enough to hold in memory. Position in the list is the only marker of how far the operator has got.
  • Verify after doing, not instead of doing. Under challenge-response the operator configures the system from memory and then reads the list to confirm each item. Under read-do the list is the procedure itself, and once the sequence is interrupted a skipped item passes unnoticed.
  • Expect production pressure to degrade use. The observed failure was crews calling several items as one group and answering in a batch, restoring the dependence on memory the list exists to remove.

Automate every item a machine can decide

An item a program can decide does not belong on a human's list. "No console.log left in the diff" is a linter rule, "the migration is reversible" a CI job, "the ticket has an owner" a required template field. Every item moved off the list is one the operator can no longer forget.

Automation cannot absorb items needing judgment (is the rollback plan executable, or only written down?), items needing action outside the machine (support has been told, the maintenance window is announced), or items where the measurement is automatic but the decision is not (the dashboard shows a 30% traffic drop — is that expected at this hour?).

CIS Benchmarks and DISA STIGs began as documents an administrator worked through by hand; most of their items are now machine-checkable content that OpenSCAP or kube-bench evaluates against a host or a cluster.

Tools: the checklists already in use

Checklist Point of use What it holds
OWASP ASVS Security requirement review Numbered verification requirements at three levels of rigor, written to be cited individually in a contract or a test plan
SRE launch coordination checklist Before a service takes production traffic Google's questions on architecture and dependencies, capacity planning, failure modes, and rollout planning
STRIDE Design review Six threat categories walked per component
OWASP Top 10 for LLM Applications Agent and LLM feature review Ten named risk categories; a coverage prompt, not a measurement
Requirements quality characteristics Specification draft Ambiguity, inconsistency, incompleteness, unmeasurable requirements — the rule set a requirements engineer works a draft against
Pull-request template, definition of done Every change Team-authored; whatever automation has not yet absorbed
Incident runbook During an outage Ordered steps for a diagnosis or a recovery

When to use, when not

Use when the procedure repeats, its steps are known, omission is costly, and the people running it work under interruption — deployment, failover, release, incident response, production-readiness review.

Do not use it as a defect-finding aid. Handing a reviewer a checklist of defect categories to read an artifact against is the use of checklists that has been measured in software, and it did not help.

Do not mandate it without implementing it. A published list nobody was trained on and nobody owns produces compliance and no change in outcome.

Evidence

No measured benefit in code inspection. Hatton analyzed 308 inspections carried out in workshops for industrial engineers and found no evidence that checklists made a significant difference to faults found per inspection hour, whether or not the inspector was experienced (Hatton 2008)2. The same data showed individual performance varying by a factor of 10, with individuals finding about 53% of the faults and two-person teams 76% — who inspects dominates whether they hold a list.

Checklist reading is no better than ad hoc reading. In a randomized experiment, 48 graduate students in 16 three-person teams inspected two requirements specifications using ad hoc, checklist, or scenario-based methods. Checklist reviewers were no more effective than ad hoc reviewers, while scenario reviewers — each given a different systematic procedure aimed at a specific fault class — had a higher fault detection rate (Porter et al. 1995)3. Student subjects, so the ordering of the three methods is the finding, not the size of the gap.

Large effects on omission-heavy clinical procedures. The WHO's 19-item surgical safety checklist was introduced at eight hospitals in eight countries; across 3733 patients before and 3955 after, the death rate fell from 1.5% to 0.8% and inpatient complications from 11.0% to 7.0% (Haynes et al. 2009)4. Before-and-after within each site, not randomized. A five-item central-line insertion checklist — hand washing, full-barrier precautions, chlorhexidine, avoiding the femoral site, removing unnecessary catheters — across 103 Michigan ICUs and 375,757 catheter-days cut the median catheter-related bloodstream infection rate from 2.7 to 0 per 1000 catheter-days at three months, with the mean still down from 7.7 to 1.4 at 16 to 18 months (Pronovost et al. 2006)5.

Mandating the list is not implementing it. Ontario made surgical safety checklists policy across its acute care hospitals. Comparing 109,341 procedures before adoption with 106,370 after at 101 hospitals, adjusted mortality moved from 0.71% to 0.65% (odds ratio 0.91, P=0.13) and adjusted complications from 3.86% to 3.82% (P=0.29) — neither significant (Urbach et al. 2014)6. The hospitals in the WHO study joined a program and collected outcome data prospectively; Ontario's received a policy.

The transfer to software is unmeasured. The clinical results are evidence for checklists over procedures whose steps are physical, enumerable, and omission-dominated, which is the shape a deployment or a failover also has. Whether the effect carries over has not been measured. The software studies tested a different use — reading code or a specification to find unknown defects — and found nothing.

Classification

  • Quality dimensions: Reliability, Security, Functionality.
  • Area: Repeated operational procedures with known steps and costly omissions — deployment, failover, release, incident response, production-readiness review, security configuration.

Referenced by

References

  1. Degani, Asaf, and Earl L. Wiener. 1993. "Cockpit Checklists: Concepts, Design, and Use." Human Factors: The Journal of the Human Factors and Ergonomics Society 35 (2): 345–59. https://doi.org/10.1177/001872089303500209. ↩↩

  2. Hatton, Les. 2008. "Testing the Value of Checklists in Code Inspections." IEEE Software 25 (4): 82–88. https://doi.org/10.1109/MS.2008.100. ↩

  3. Porter, Adam A., Lawrence G. Votta, and Victor R. Basili. 1995. "Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment." IEEE Transactions on Software Engineering 21 (6): 563–75. https://doi.org/10.1109/32.391380. ↩

  4. Haynes, Alex B., Thomas G. Weiser, William R. Berry, et al. 2009. "A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population." New England Journal of Medicine 360 (5): 491–99. https://doi.org/10.1056/NEJMsa0810119. ↩

  5. Pronovost, Peter, Dale Needham, Sean Berenholtz, et al. 2006. "An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU." New England Journal of Medicine 355 (26): 2725–32. https://doi.org/10.1056/NEJMoa061115. ↩

  6. Urbach, David R., Anand Govindarajan, Refik Saskin, Andrew S. Wilton, and Nancy N. Baxter. 2014. "Introduction of Surgical Safety Checklists in Ontario, Canada." New England Journal of Medicine 370 (11): 1029–38. https://doi.org/10.1056/NEJMsa1308261. ↩