# Reliability

Failure rate over time; behavior under partial failure. The dimension
that separates demos from production systems. Reliability concerns the
unhappy paths — fault propagation, recovery — not the common-case
behavior functional tests cover.

## What goes wrong

- **Partial failures.** A disk fails halfway through a write. A
  network partition splits the cluster mid-election. One replica
  diverges silently. These cannot be enumerated by hand.
- **Data loss or corruption under race conditions.** Two writers
  racing; one wins, the other's data is lost. Visible only at scale
  or under stress.
- **Cascading failures.** One service slows, its callers retry,
  retries amplify load, downstream falls over. Self-reinforcing.
- **Recovery bugs.** The system stays up but doesn't fully recover
  after a fault. Discovered weeks later when a second fault
  exposes that the first wasn't healed.
- **Byzantine-shaped bugs.** Clock skew, GC pauses, network
  reordering. Behavior the protocol designer didn't model.
- **Heisenbugs.** Non-reproducible. Two minutes of failure once a
  month. Each costs engineer-weeks of debugging.
- **Deploy-time schema-migration downtime.** A migration takes a lock
  that stalls reads and writes, or a backward-incompatible change breaks
  the code still running through a rolling deploy.

## What success looks like

- **Recovery is complete.** The system fully recovers after a fault,
  not merely survives, and knows and reports when it has.
- **Mean time to recover is shorter than mean time between
  failures, deliberately.** The design optimizes for recovery speed
  over failure prevention.
- **Every postmortem closes with an SLI or invariant.** *"If we'd
  alerted on metric X, we would have caught this 12 minutes
  earlier."* That alert exists by the next incident.
- **The rare bug is reproducible.** If DST or replay catches it,
  the diff is small and the reproducer is bit-exact.
- **Engineering SLOs are tighter than the customer-facing SLAs,
  not the other way around.** Headroom is engineered, not
  promised away.

## Referenced by

- [Quality dimensions](https://quality.stereobooster.com/quality-dimensions.md) · Quality dimensions
- [Choosing methods](https://quality.stereobooster.com/choosing.md) · Overview

## Acronyms

- DST — deterministic simulation testing
- SLA — service level agreement
- SLI — service level indicator
- SLO — service level objective