# Chaos engineering Chaos engineering injects faults into a production-like or production system and verifies that the system survives. A fault kills a process, slows a network, exhausts a disk, or drops a region. Netflix's article defines it as "the discipline of experimenting on a distributed system in order to build confidence in its capability to withstand turbulent conditions in production" (Basiri et al. 2016)[^basiri2016]; a fallback that has never run supplies no such confidence. The technique is the operational counterpart to [deterministic simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md): DST simulates the faults; chaos engineering injects them into the running system. ## What it catches - **Hidden dependencies.** Service A claims to gracefully degrade when B is down; the chaos run shows it times out and returns 500. - **Untested fallbacks.** Cache-miss paths, retry budgets, circuit breakers, timeout cascades. Such code has run only in tests, never in production. - **Capacity limits and saturation.** Killing one instance of a pool reveals whether the rest absorb the load or fail in sequence. - **Region- and zone-level failures.** Pulling the plug on us-east-1 exercises a failover plan that has never run. - **Clock-skew bugs.** Clock-skew injection finds distributed-consensus bugs that appear only when nodes disagree about the time: split-brain, lost writes, stale reads. - **Recovery-time assumptions.** A promised five-minute failover becomes a measured failover time. Chaos engineering does not catch correctness bugs that never manifest as availability failures, nor bugs that require an interleaving the experiment never generated; [deterministic simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md) reaches those. ## Tools ### Fault injection at the orchestrator level - **LitmusChaos** — Kubernetes-native; CNCF graduated. - **Chaos Mesh** — Kubernetes-native; CNCF incubation. - **Gremlin** — commercial; hosted fault injection. - **AWS Fault Injection Service**, **Azure Chaos Studio** — cloud-provider managed. ### Network-fault injection - **[Toxiproxy](https://github.com/Shopify/toxiproxy)** (Shopify) — TCP proxy that injects latency, packet loss, partitions, bandwidth caps. - **[Pumba](https://github.com/alexei-led/pumba)** — Docker fault injection (kill, netem-based latency and loss). - **[tc](https://man7.org/linux/man-pages/man8/tc.8.html)** (Linux Traffic Control), **[netem](https://man7.org/linux/man-pages/man8/tc-netem.8.html)** — the underlying kernel primitives. ### Application-level chaos - **Chaos Toolkit** — language-agnostic experiment runner. - **OpenChaos** — protocol-level chaos for distributed systems. ### Adjacent / open-source production-testing tools - **[Jepsen](https://jepsen.io/)** (Kingsbury) — black-box concurrency testing against real databases; the public record on distributed-system bugs. ## When to use, when not **Use:** - Production systems where availability is a quality dimension. The cheapest entry point is staging-environment fault injection; graduate to production with strict blast-radius limits once staging is clean. - Microservice topologies where fallbacks were declared but never exercised. The hidden-dependency failure mode is endemic there. - Multi-region or multi-zone deployments. Zone-loss handling is configuration until a zone is lost. - GameDays and on-call training. A scheduled experiment tests the responders and the runbook, not only the system. **Don't:** - On systems that are not yet observable. Without [monitoring](https://quality.stereobooster.com/monitoring-and-observability.md), a fault the experiment caused cannot be told apart from one it exposed. - Without an explicit blast-radius plan. "Production chaos" without a stop button is just an outage. - Instead of [DST](https://quality.stereobooster.com/deterministic-simulation-testing.md) when the goal is bug-finding rather than fallback verification. DST reaches deeper at lower cost; chaos catches the hidden-dependency and runbook-correctness bugs DST does not. ## Evidence - **The concrete bug record.** Kingsbury's public Jepsen analyses document real consistency and consensus bugs found by black-box fault injection against MongoDB, CockroachDB, etcd, PostgreSQL replication, RabbitMQ, and many more (Kingsbury n.d.)[^kingsburynd]. Kingsbury does not call it chaos engineering, but the mechanism is the same. - **Framing and experience, not a trial.** Netflix's IEEE Software article states the principles and reports their own practice (Basiri et al. 2016)[^basiri2016]; the *Chaos Engineering* report is the long-form practitioner treatment (Rosenthal et al. 2017)[^rosenthal2017]. Neither is a controlled study. The empirical literature is industrial, not academic: no controlled with-chaos-versus-without trial exists. ## Related **Perturbation testing: what you perturb, and where** All three deliberately subject a system to bad conditions, differing in what they perturb and where. [Fuzzing](https://quality.stereobooster.com/fuzzing.md) generates *inputs* (random, coverage-guided, or grammar-aware) and watches for crashes, undefined behavior, or sanitizer trips. [Fault injection](https://quality.stereobooster.com/fault-injection.md) perturbs the *environment* in a test harness (a failed allocation, a dropped packet, a crashed dependency) to check the error-handling and fault-tolerance paths. Chaos engineering injects the same kind of environment faults into a *running production* system, verifying fallbacks under real conditions. Fuzzing probes input handling; fault injection and chaos probe failure resilience, in a harness and in production respectively. ## Classification - **Quality dimensions:** Reliability, Performance. - **Area:** Microservices, multi-region and multi-zone deployments, on-call operations, GameDays. - **Guarantee:** Empirical. ## Referenced by - [Quality dimensions](https://quality.stereobooster.com/quality-dimensions.md) · Quality dimensions - [Effect scope](https://quality.stereobooster.com/effect.md) · The axes - [Fault injection](https://quality.stereobooster.com/fault-injection.md) · Methods - [Snapshot and approval testing](https://quality.stereobooster.com/snapshot-testing.md) · Methods - [Verifying concurrency](https://quality.stereobooster.com/concurrency.md) · Methods - [How AI fits into software quality](https://quality.stereobooster.com/ai.md) · AI ## References [^basiri2016]: Basiri, Ali, Niosha Behnam, Ruud de Rooij, et al. 2016. "[Chaos Engineering](https://arxiv.org/pdf/1702.05843)." *IEEE Software* 33 (3): 35–41. . [^kingsburynd]: Kingsbury, Kyle. n.d. *[Jepsen: Distributed Systems Safety Analyses](https://jepsen.io/analyses)*. Jepsen.io. . [^rosenthal2017]: Rosenthal, Casey, Lorin Hochstein, Aaron Blohowiak, Nora Jones, and Ali Basiri. 2017. *[Chaos Engineering](https://www.oreilly.com/content/chaos-engineering/)*. O'Reilly Media. . ## Acronyms - DST — deterministic simulation testing