Skip to content

Software Quality

Fault injection

Fault injection deliberately triggers the failures a system is meant to tolerate, to check that the code written to handle them actually does. The injected faults include a failed allocation, a dropped packet, a corrupted record, a crashed dependency.

The technique spans levels. Chaos engineering injects faults at the system level on a production-like system, and deterministic simulation testing injects them inside a deterministic simulator. In-process injection forces one chosen internal operation to fail and exercises the error path that follows. It needs neither an orchestrator nor a simulator, which makes it the cheapest of the three to aim at one specific error path.

What it catches

  • Untested error-handling paths. The catch block never entered in testing, the retry that deadlocks, the cleanup that leaks a resource when the operation it follows fails partway.
  • Resource-exhaustion handling. Whether the code degrades or corrupts under out-of-memory, disk-full or descriptor exhaustion — a failed malloc, a write erroring mid-stream.
  • Partial-failure handling. An RPC or syscall that fails after a side effect, leaving state half-updated; the bug is in what happens next.
  • Transient hardware-fault tolerance. For safety-critical systems, software-implemented fault injection (SWIFI) corrupts registers and memory to test survival of bit-flips a normal test never produces.

What it does not catch: anything on the happy path, and faults outside the injected set, which the fault model bounds.

Tools

  • libfiu — intercepts libc calls to fail them on demand (C/C++).
  • failpoints — compile-time injection points toggled at runtime: the Rust fail crate, FreeBSD's fail(9).
  • Byteman (Java) — rules that inject exceptions or delays at named code points.
  • Toxiproxy — injects network faults (latency, drops) at a proxy, sitting between the in-process and system levels.

When to use, when not

Use it where error-handling or fault-tolerance code is critical and hard to trigger naturally — storage engines, allocators, network clients, retry and failover logic, safety-critical software.

Don't treat it as a substitute for happy-path testing, and don't trust a fault model that does not match real failure modes: injecting faults that cannot happen wastes effort, and missing the ones that can gives false confidence.

Evidence

The case is definitional rather than measured: error-handling code that never runs under normal load is untested code, and injecting the fault is the way to run it.

Further reading

  • The technique and its hardware-vs-software taxonomy are surveyed by Hsueh, Tsai, and Iyer (Hsueh et al. 1997)1.

Perturbation testing: what you perturb, and where

All three deliberately subject a system to bad conditions, differing in what they perturb and where. Fuzzing generates inputs (random, coverage-guided, or grammar-aware) and watches for crashes, undefined behavior, or sanitizer trips. Fault injection perturbs the environment in a test harness (a failed allocation, a dropped packet, a crashed dependency) to check the error-handling and fault-tolerance paths. Chaos engineering injects the same kind of environment faults into a running production system, verifying fallbacks under real conditions. Fuzzing probes input handling; fault injection and chaos probe failure resilience, in a harness and in production respectively.

Classification

  • Quality dimensions: Reliability: safety.
  • Area: Error-handling and fault-tolerance code that normal runs rarely exercise — storage and allocator error paths, RPC partial failures, retry and cleanup logic; safety-critical tolerance to transient hardware faults.
  • Guarantee: Empirical — the faults that were injected were handled; faults outside the injected set say nothing.

Referenced by

References

  1. Hsueh, Mei-Chen, Timothy K. Tsai, and Ravishankar K. Iyer. 1997. "Fault Injection Techniques and Tools." Computer 30 (4): 75–82. https://doi.org/10.1109/2.585157. ↩