# Fault injection

Fault injection deliberately triggers the failures a system is meant
to tolerate, to check that the code written to handle them actually
does. The injected faults include a failed allocation, a dropped
packet, a corrupted record, a crashed dependency.

The technique spans levels. [Chaos engineering](https://quality.stereobooster.com/chaos-engineering.md)
injects faults at the **system** level on a production-like system, and
[deterministic simulation testing](https://quality.stereobooster.com/deterministic-simulation-testing.md)
injects them inside a **deterministic simulator**. **In-process**
injection forces one chosen internal operation to fail and exercises
the error path that follows. It needs
neither an orchestrator nor a simulator, which makes it the
cheapest of the three to aim at one specific error path.

## What it catches

- **Untested error-handling paths.** The catch block never entered in
  testing, the retry that deadlocks, the cleanup that leaks a resource
  when the operation it follows fails partway.
- **Resource-exhaustion handling.** Whether the code degrades or
  corrupts under out-of-memory, disk-full or descriptor exhaustion —
  a failed `malloc`, a write erroring mid-stream.
- **Partial-failure handling.** An RPC or syscall that fails after a
  side effect, leaving state half-updated; the bug is in what happens
  next.
- **Transient hardware-fault tolerance.** For safety-critical systems,
  software-implemented fault injection (SWIFI) corrupts registers and
  memory to test survival of bit-flips a normal test never produces.

What it does not catch: anything on the happy path, and faults
outside the injected set, which the fault model bounds.

## Tools

- **[libfiu](https://blitiri.com.ar/p/libfiu/)** — intercepts libc calls to fail them on
  demand (C/C++).
- **failpoints** — compile-time injection points toggled at runtime:
  the Rust `fail` crate, FreeBSD's `fail(9)`.
- **Byteman** (Java) — rules that inject exceptions or delays at named
  code points.
- **[Toxiproxy](https://github.com/Shopify/toxiproxy)** — injects network faults (latency,
  drops) at a proxy, sitting between the in-process and system levels.

## When to use, when not

**Use** it where error-handling or fault-tolerance code is critical and
hard to trigger naturally — storage engines, allocators, network
clients, retry and failover logic, safety-critical software.

**Don't** treat it as a substitute for happy-path testing, and don't
trust a fault model that does not match real failure modes: injecting
faults that cannot happen wastes effort, and missing the ones that can
gives false confidence.

## Evidence

The case is definitional rather than measured: error-handling code
that never runs under normal load is untested code, and injecting the
fault is the way to run it.

## Further reading

- The technique and its hardware-vs-software taxonomy are surveyed
  by Hsueh, Tsai, and Iyer (Hsueh et al. 1997)[^hsueh1997].

## Related

**Perturbation testing: what you perturb, and where**

All three deliberately subject a system to bad conditions, differing in what
they perturb and where. [Fuzzing](https://quality.stereobooster.com/fuzzing.md) generates *inputs*
(random, coverage-guided, or grammar-aware) and watches for crashes, undefined
behavior, or sanitizer trips. Fault injection
perturbs the *environment* in a test harness (a failed allocation, a dropped
packet, a crashed dependency) to check the error-handling and fault-tolerance
paths. [Chaos engineering](https://quality.stereobooster.com/chaos-engineering.md) injects the same
kind of environment faults into a *running production* system, verifying
fallbacks under real conditions. Fuzzing probes input handling; fault injection
and chaos probe failure resilience, in a harness and in production respectively.

## Classification

- **Quality dimensions:** Reliability: safety.
- **Area:** Error-handling and fault-tolerance code that normal runs rarely exercise — storage and allocator error paths, RPC partial failures, retry and cleanup logic; safety-critical tolerance to transient hardware faults.
- **Guarantee:** Empirical — the faults that were injected were handled; faults outside the injected set say nothing.

## Referenced by

- [Snapshot and approval testing](https://quality.stereobooster.com/snapshot-testing.md) · Methods
- [Verifying memory safety](https://quality.stereobooster.com/memory.md) · Methods
- [Verifying safety-critical systems](https://quality.stereobooster.com/safety.md) · Methods

## References

[^hsueh1997]: Hsueh, Mei-Chen, Timothy K. Tsai, and Ravishankar K. Iyer. 1997. "[Fault Injection Techniques and Tools](https://gse.ufsc.br/bezerra/disciplinas/Confiabilidade/docs/hsueh-injection.pdf)." *Computer* 30 (4): 75–82. <https://doi.org/10.1109/2.585157>.

## Acronyms

- SWIFI — software-implemented fault injection