Skip to content

Software Quality

Reliability

Does it keep working when parts of it fail?

Failure rate over time; behavior under partial failure. The dimension that separates demos from production systems. Reliability concerns the unhappy paths — fault propagation, recovery — not the common-case behavior functional tests cover.

What goes wrong

  • Partial failures. A disk fails halfway through a write. A network partition splits the cluster mid-election. One replica diverges silently. These cannot be enumerated by hand.
  • Data loss or corruption under race conditions. Two writers racing; one wins, the other's data is lost. Visible only at scale or under stress.
  • Cascading failures. One service slows, its callers retry, retries amplify load, downstream falls over. Self-reinforcing.
  • Recovery bugs. The system stays up but doesn't fully recover after a fault. Discovered weeks later when a second fault exposes that the first wasn't healed.
  • Byzantine-shaped bugs. Clock skew, GC pauses, network reordering. Behavior the protocol designer didn't model.
  • Heisenbugs. Non-reproducible. Two minutes of failure once a month. Each costs engineer-weeks of debugging.
  • Deploy-time schema-migration downtime. A migration takes a lock that stalls reads and writes, or a backward-incompatible change breaks the code still running through a rolling deploy.

What success looks like

  • Recovery is complete. The system fully recovers after a fault, not merely survives, and knows and reports when it has.
  • Mean time to recover is shorter than mean time between failures, deliberately. The design optimizes for recovery speed over failure prevention.
  • Every postmortem closes with an SLI or invariant. "If we'd alerted on metric X, we would have caught this 12 minutes earlier." That alert exists by the next incident.
  • The rare bug is reproducible. If DST or replay catches it, the diff is small and the reproducer is bit-exact.
  • Engineering SLOs are tighter than the customer-facing SLAs, not the other way around. Headroom is engineered, not promised away.

Referenced by