Skip to content

Software Quality

Fuzzing

Fuzzing is an automated testing method that runs a program on generated inputs and reports a crash, undefined behavior or a sanitizer violation as a failure. Nothing has to state the expected output for any given input, and that is what lets the input count run into the millions.

What it catches

Fuzzing is most effective on memory-safety and crash bugs. With the right oracles attached (sanitizers, invariants), it extends to logic bugs.

  • Memory-safety bugs. Buffer overflows, use-after-free, double-free, out-of-bounds reads. AddressSanitizer makes these observable.
  • Undefined behavior in C/C++. UndefinedBehaviorSanitizer flags signed overflow, null deref, invalid shifts.
  • Concurrency bugs. ThreadSanitizer catches data races; thread-aware fuzzers (Krace, MUZZ) steer the search into them.
  • Logic bugs surfaced by invariants. If the code asserts invariants, fuzzing exercises them across an enormous input space.
  • Parser bugs. Crashes, infinite loops, exponential-time inputs (CVE pattern; ReDoS for regex).
  • Protocol bugs. State-machine fuzzers (e.g., AFLNet, syzkaller) catch deep protocol violations.

Fuzzing does not catch silent wrong-output bugs unless you attach an oracle. A fuzzer that finds no crashes has not shown the program correct.

Tools

Continuous fuzzing infrastructure

  • OSS-Fuzz — Google's continuous fuzzing service for open source. Free. Required for security-sensitive open-source C/C++ that handles untrusted input.

Coverage-guided fuzzers

  • libFuzzer — in-process, fast; ships with LLVM. Best for C and C++ APIs.
  • AFL++ — process-based; the AFL successor; standalone binaries.
  • Honggfuzz — similar trade-offs; alternative to AFL.

Per ecosystem

Specialized fuzzers

  • syzkaller — Linux kernel syscall fuzzer.
  • AFLNet — protocol-aware fuzzing.
  • Csmith — generates random C programs; the canonical compiler-fuzzing example.
  • SQLancer — SQL fuzzing with metamorphic oracles for SQLite, MySQL, and PostgreSQL (Rigger and Su 2020)1.
  • PerfFuzz / SlowFuzz — performance fuzzers: evolve inputs to maximize execution cost rather than to trigger a crash, surfacing worst-case algorithmic blow-ups (algorithmic complexity testing).

Concurrency fuzzers

Ordinary coverage feedback tracks which code a run reaches, not which interleaving — so a gray-box fuzzer can hammer multithreaded code without ever varying the schedule. These add interleaving to the feedback loop:

  • Krace — coverage-guided data-race fuzzer for kernel file systems; it adds an interleaving-coverage signal alongside branch coverage, so the search explores thread schedules and not just inputs (Xu et al. 2020)2.
  • MUZZ — thread-aware gray-box fuzzing for multithreaded programs; instruments to stress the thread-interleaving states ordinary coverage ignores (Chen et al. 2020)3.

These chase the same interleaving bugs as systematic concurrency testing, but by coverage-guided sampling rather than systematic or PCT-bounded exploration.

Curated input corpora

  • Big List of Naughty Strings — a community-curated list of strings that have historically broken parsers and validators (Unicode edge cases, RTL marks, format-string syntax, SQL injection patterns, very long strings). Useful as a starter corpus for string-handling fuzz harnesses and for property-test generators that need realistic edge cases.

When to use, when not

Use:

  • Any code that parses untrusted input (network protocols, file formats, deserialization, regex compilation).
  • Security-sensitive C/C++ code regardless of input source.
  • Open-source libraries that handle parsing or crypto — submit to OSS-Fuzz.
  • Linux kernel paths or syscall interfaces — syzkaller.

Don't:

  • As your only verification. Fuzzing finds crashes; logic bugs need oracles you supply (properties, differential references, invariants).
  • For pure business logic with no crash risk. The marginal value is low compared to property tests.
  • For UI code, where crashes are caught by the platform and the failure modes are visual.

Running a productive campaign

  • Mutate at semantic boundaries, not raw bytes. Blind bit-flips and truncation die at the first validity check and never reach interesting code. The bugs come from handing the target a plausible-but-wrong structure it will follow — corrupt a length or offset field, a record count, a type tag: values the parser trusts. This is the case for structure-aware fuzzing: encode the format so the fuzzer mutates fields, not bytes (Google n.d.)4; see also FormatFuzzer (Dutra et al. 2021)5.
  • Make the contract strict — "typed error or clean," never just "didn't crash." "The process stayed up" is too weak an oracle: it passes a parser that leaks a raw RangeError / NullPointerException to its caller, or one that hangs. Assert instead that every rejection is the library's own typed error and that nothing hangs. A raw runtime error or a timeout becomes a finding, not a pass — which is what turns a crash fuzzer into a robustness fuzzer without a full differential oracle.
  • Iterate by layer: fix a class → re-fuzz → next layer. You cannot see bug N+1 until bug N stops throwing. A binary reader peels in layers — bounds error, then over-allocation, then a corrupt count driving a huge array, then a view over a short slice — each hidden behind the previous. Budget several rounds.
  • Seed diversely. A handful of structurally different valid inputs reaches far more code than one input mutated repeatedly.
  • Profile the oracle, not the RNG. When a campaign is slow the bottleneck is almost always per-iteration work — building the input or an expensive oracle — not input generation. An oracle that materializes a large result the property never inspects can dominate wall-clock; bounding the inputs so the oracle stays cheap can cut runtime by an order of magnitude at the same coverage. Profiling the phases comes before parallelizing them.
  • Keep a fast gate and a deep campaign; don't consolidate. A property-based generator (fast-check, Hypothesis) gives a structured oracle and shrinks a failure to a minimal counterexample — run it with a fixed seed as an always-on CI gate (seconds). A coverage-guided byte fuzzer (libFuzzer, Jazzer/Jazzer.js, AFL++) steers into deep branches blind mutation can't reach — run it as a non-deterministic campaign against a corpus and a time budget. Neither subsumes the other.
  • The harness is the asset, not the fix list. A list of bugs fixed is a snapshot; the harness keeps finding the next layer and guards against regressions. Keep it in the repo and raise the run count on a schedule. Fuzz both doors where untrusted input crosses in: the read side (arbitrary bytes) and the write side (arbitrary input → builder, then a build→read round-trip property).

Recurring bug classes in parsers and binary readers

  • Sequential reads off the end. Bound the shared read primitive once, so an out-of-bounds read throws the typed error, not a raw RangeError. Wrapping each call site individually is what creates the gap; enforce the bound where bytes become values.
  • Allocation sized by an attacker-controlled length word. A corrupt length field drives an enormous allocation — and an in-range multi-GB request succeeds before it OOMs, so an "is it allocatable at all" guard misses it. Clamp the read to the known end of input at the chokepoint rather than trusting the word. For speculative/prefetch reads, clamp (resolve truncated) rather than throw, so a fire-and-forget read can't reject unhandled; surface the corruption typed on the awaited path.
  • Sentinel-terminated decode loops. A loop that scans "until the terminating bit/byte" runs forever past a truncated region (the out-of-range read returns a value that never matches the sentinel) → a single input hangs the process → denial of service. Bound every such loop by a counted length, never the sentinel alone. Grep byte-stream code for for (;;) / while (true).
  • Lazy "ensure-then-read" windows. An accessor that assumes a prior ensure step ran must still bounds-check its index against the real count — corruption reaches the read without the ensure. Every such accessor needs a typed corruption guard.
  • Cross-structure invariants are reachable by corruption. When the set a read assumes was prepared comes from a different on-disk structure than its read key, corrupt bytes can desync the two, firing an internal "you must call X first" assertion (meant as a programming-bug signal) on data corruption. Give those guards the typed corruption error too, not a bare Error.
  • Every corrupt-input guard must throw the library's typed error. A guard that throws a bare Error reads as a crash to the harness and to the caller. One typed error type at the deserialize boundary fixes most of these classes at once.

Evidence

  • OSS-Fuzz public counters (May 2025): >13,000 vulnerability fixes, >50,000 bug fixes across >1,000 projects (Google 2025)6.
  • syzkaller. One independent analysis of 125,000+ kernel vulnerabilities found that the share of Linux kernel bugs fixed within a year of being introduced rose from ~0% (2010) to ~69% (2022), attributing the shift to syzkaller plus KASAN/KMSAN/KCSAN (Qu 2026)7; the primary source is a blog analysis of kernel git history, not a peer-reviewed study — the direction of the effect is well-attested, the exact percentages should be treated as indicative.
  • Csmith found 325+ unknown compiler bugs in GCC, LLVM, and others over three years (Yang et al. 2011)8.
  • The CompCert outlier: Csmith found no wrong-code bugs in its formally verified middle-end (Leroy 2009; Yang et al. 2011)9 8 — the contrast that bounds what fuzzing alone reaches.

Exploring the program's behavior space

All explore many program behaviors; they differ in what they run and what they guarantee. Model checking works on a spec or model and is exhaustive over it. Symbolic execution runs the actual code, solving path constraints to reach branches a fuzzer can't, with a bounded-exhaustive verdict; bounded model checking runs the actual code too, but poses a single SMT query over the whole program unrolled to a fixed depth rather than exploring paths one at a time. Fuzzing runs the actual code too but samples inputs empirically: cheaper per case, no exhaustiveness claim.

Perturbation testing: what you perturb, and where

All three deliberately subject a system to bad conditions, differing in what they perturb and where. Fuzzing generates inputs (random, coverage-guided, or grammar-aware) and watches for crashes, undefined behavior, or sanitizer trips. Fault injection perturbs the environment in a test harness (a failed allocation, a dropped packet, a crashed dependency) to check the error-handling and fault-tolerance paths. Chaos engineering injects the same kind of environment faults into a running production system, verifying fallbacks under real conditions. Fuzzing probes input handling; fault injection and chaos probe failure resilience, in a harness and in production respectively.

Generative testing

These methods blur together because they all run the code and check the result without a hand-written expected value. Two roles pull them apart, and a single test picks one of each.

Input generators choose the input, and differ by the steering signal (the Generative subtree of the input axis):

Oracle suppliers provide the verdict when no expected value is written:

Pick one generator and one oracle: they compose. A coverage-guided fuzzer that checks a metamorphic relation is fuzzing and metamorphic at once. (The written-answer end, where the author picks rows and answers by hand, is example / parameterized tests.)

Classification

  • Quality dimensions: Security, Functionality, Reliability.
  • Area: Parsers, security-sensitive C/C++, kernels and syscall surfaces, network protocols, crypto, compression and image decoders.
  • Guarantee: Empirical.

Referenced by

References

  1. Rigger, Manuel, and Zhendong Su. 2020. "Testing Database Engines via Pivoted Query Synthesis." Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI '20). https://doi.org/10.48550/arXiv.2001.04174. ↩

  2. Xu, Meng, Sanidhya Kashyap, Hanqing Zhao, and Taesoo Kim. 2020. "Krace: Data Race Fuzzing for Kernel File Systems." 2020 IEEE Symposium on Security and Privacy (SP), 1643–60. https://doi.org/10.1109/SP40000.2020.00078. ↩

  3. Chen, Hongxu, Shengjian Guo, Yinxing Xue, et al. 2020. "MUZZ: Thread-aware Grey-box Fuzzing for Effective Bug Hunting in Multithreaded Programs." 29th USENIX Security Symposium (USENIX Security 20), 2325–42. https://doi.org/10.48550/arXiv.2007.15943. ↩

  4. Google. n.d. Structure-Aware Fuzzing with libFuzzer. Google/fuzzing. https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md. ↩

  5. Dutra, Rafael, Rahul Gopinath, and Andreas Zeller. 2021. FormatFuzzer: Effective Fuzzing of Binary File Formats. https://doi.org/10.48550/arXiv.2109.11277. ↩

  6. Google. 2025. OSS-Fuzz: Continuous Fuzzing for Open Source Software. Google/oss-fuzz, GitHub. https://github.com/google/oss-fuzz. ↩

  7. Qu, Jenny Guanni. 2026. Kernel Bugs Hide for 2 Years on Average. Some Hide for 20. Pebblebed Blog. https://pebblebed.com/blog/kernel-bugs. ↩

  8. Yang, Xuejun, Yang Chen, Eric Eide, and John Regehr. 2011. "Finding and Understanding Bugs in C Compilers." Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '11), 283–94. https://doi.org/10.1145/1993498.1993532. ↩↩

  9. Leroy, Xavier. 2009. "Formal Verification of a Realistic Compiler." Communications of the ACM 52 (7): 107–15. https://doi.org/10.1145/1538788.1538814. ↩