Chaos engineering injects faults into a production-like or production system and verifies that the system survives. A fault kills a process, slows a network, exhausts a disk, or drops a region. Netflix's article defines it as "the discipline of experimenting on a distributed system in order to build confidence in its capability to withstand turbulent conditions in production" (Basiri et al. 2016)¹; a fallback that has never run supplies no such confidence. The technique is the operational counterpart to deterministic simulation testing: DST simulates the faults; chaos engineering injects them into the running system.

What it catches

• Hidden dependencies. Service A claims to gracefully degrade when B is down; the chaos run shows it times out and returns 500.
• Untested fallbacks. Cache-miss paths, retry budgets, circuit breakers, timeout cascades. Such code has run only in tests, never in production.
• Capacity limits and saturation. Killing one instance of a pool reveals whether the rest absorb the load or fail in sequence.
• Region- and zone-level failures. Taking us-east-1 offline exercises a failover plan that has never run.
• Clock-skew bugs. Clock-skew injection finds distributed-consensus bugs that appear only when nodes disagree about the time: split-brain, lost writes, stale reads.
• Recovery-time assumptions. A promised five-minute failover becomes a measured failover time.

Chaos engineering does not catch correctness bugs that never manifest as availability failures, nor bugs that require an interleaving the experiment never generated; deterministic simulation testing reaches those.

Tools

Fault injection at the orchestrator level

• LitmusChaos — Kubernetes-native; CNCF incubating.
• Chaos Mesh — Kubernetes-native; CNCF incubating.
• Gremlin — commercial; hosted fault injection.
• AWS Fault Injection Service, Azure Chaos Studio — commercial; cloud-provider managed.

Network-fault injection

• Toxiproxy (Shopify) — TCP proxy that injects latency, packet loss, partitions, bandwidth caps.
• Pumba — Docker fault injection (kill, netem-based latency and loss).
• tc (Linux Traffic Control), netem — the underlying kernel primitives.

Application-level chaos

• Chaos Toolkit — language-agnostic experiment runner.
• OpenChaos — protocol-level chaos for distributed systems.

Adjacent tools

• Jepsen (Kingsbury) — black-box concurrency testing against real databases; the public record on distributed-system bugs.

When to use, when not

Use:

• Production systems with an availability requirement. The cheapest entry point is staging-environment fault injection; graduate to production with strict blast-radius limits once staging is clean.
• Microservice topologies where fallbacks were declared but never exercised. The hidden-dependency failure mode is endemic there.
• Multi-region or multi-zone deployments. Zone-loss handling is configuration until a zone is lost.
• GameDays and on-call training. A scheduled experiment tests the responders and the runbook, not only the system.

Don't:

• On systems that are not yet observable. Without monitoring, a fault the experiment caused cannot be told apart from one it exposed.
• Without an explicit blast-radius plan. Production chaos without a stop button is an outage.
• Instead of DST when the goal is bug-finding rather than fallback verification. DST reaches deeper; chaos catches the hidden-dependency and runbook-correctness bugs DST does not.

Evidence

• The concrete bug record. Kingsbury's public Jepsen analyses document replica divergence, data loss, stale reads and read skew found by black-box fault injection against over two dozen databases, coordination services and queues (MongoDB, CockroachDB, etcd, PostgreSQL and RabbitMQ among them) (Kingsbury n.d.)². Kingsbury does not call it chaos engineering, but the mechanism is the same.
• Framing and experience, not a trial. Netflix's IEEE Software article states the principles and reports their own practice (Basiri et al. 2016)¹; the Chaos Engineering report is the long-form practitioner treatment (Rosenthal et al. 2017)³. Neither is a controlled study.

Related

Perturbation testing: what you perturb, and where

All three deliberately subject a system to bad conditions, differing in what they perturb and where. Fuzzing generates inputs (random, coverage-guided, or grammar-aware) and watches for crashes, undefined behavior, or sanitizer trips. Fault injection perturbs the environment in a test harness (a failed allocation, a dropped packet, a crashed dependency) to check the error-handling and fault-tolerance paths. Chaos engineering injects the same kind of environment faults into a running production system, verifying fallbacks under real conditions. Fuzzing probes input handling; fault injection and chaos probe failure resilience, in a harness and in production respectively.

Classification

• Quality dimensions: Reliability, Performance.
• Area: Microservices, multi-region and multi-zone deployments, on-call operations, GameDays.
• Guarantee: Empirical.

Referenced by

• Quality dimensions · Quality dimensions
• Effect scope · The axes
• Fault injection · Methods
• Snapshot and approval testing · Methods
• Verifying concurrency · Methods
• How AI fits into software quality · AI

References

---

1. Basiri, Ali, Niosha Behnam, Ruud de Rooij, et al. 2016. "Chaos Engineering." IEEE Software 33 (3): 35–41. https://doi.org/10.1109/MS.2016.60. ↩↩

2. Kingsbury, Kyle. n.d. Jepsen: Distributed Systems Safety Analyses. Jepsen.io. https://jepsen.io/analyses. ↩

3. Rosenthal, Casey, Lorin Hochstein, Aaron Blohowiak, Nora Jones, and Ali Basiri. 2017. Chaos Engineering. O'Reilly Media. https://www.oreilly.com/content/chaos-engineering/. ↩