Skip to content

Software Quality

Fuzzing

Fuzzing is an automated testing method that runs a program on generated inputs and reports a crash, undefined behavior or a sanitizer violation as a failure. Nothing has to state the expected output for any given input, and that is what lets the input count run into the millions.

What it catches

Fuzzing is most effective on memory-safety and crash bugs.

  • Memory-safety bugs. Buffer overflows, use-after-free, double-free, out-of-bounds reads. AddressSanitizer makes these observable.
  • Undefined behavior in C/C++. UndefinedBehaviorSanitizer flags signed overflow, null deref, invalid shifts.
  • Concurrency bugs. ThreadSanitizer catches data races; thread-aware fuzzers (Krace, MUZZ) steer the search into them.
  • Logic bugs surfaced by invariants. If the code asserts invariants, fuzzing exercises them across an enormous input space.
  • Parser bugs. A malformed document crashes the parser, sends it into an infinite loop, or drives it down an exponential-time path — the CVE pattern, and ReDoS for regex.
  • Protocol bugs. State-machine fuzzers such as AFLNet catch deep protocol violations.

Fuzzing does not catch silent wrong-output bugs unless the harness attaches an oracle. A fuzzer that finds no crashes has not shown the program correct.

Tools

Continuous fuzzing infrastructure

  • OSS-Fuzz — Google runs accepted open-source projects' fuzz harnesses continuously, at no cost to the project.

Coverage-guided fuzzers

  • libFuzzer — in-process, fast; ships with LLVM. Best for C and C++ APIs.
  • AFL++ — runs the target as a separate process, so it fuzzes standalone binaries.
  • Honggfuzz — makes the same process-based trade-offs as AFL++.

Per ecosystem

Specialized fuzzers

  • syzkaller — Linux kernel syscall fuzzer, run against a kernel built with the kernel's own sanitizers, KASAN/KMSAN/KCSAN.
  • AFLNet — protocol-aware fuzzing.
  • Csmith — fuzzes C compilers by generating random C programs.
  • SQLancer — synthesizes a SQL query that must fetch a randomly chosen row and reports an engine that fails to return it, which finds logic bugs a crash oracle misses; tested on SQLite, MySQL, and PostgreSQL (Rigger and Su 2020)1.
  • PerfFuzz / SlowFuzz — performance fuzzers: evolve inputs to maximize execution cost rather than to trigger a crash, surfacing worst-case algorithmic blow-ups (algorithmic complexity testing).

Concurrency fuzzers

Ordinary coverage feedback tracks which code a run reaches, not which interleaving — so a gray-box fuzzer can run millions of inputs through multithreaded code without ever varying the schedule. These add interleaving to the feedback loop:

  • Krace — coverage-guided data-race fuzzer for kernel file systems; it adds an interleaving-coverage signal alongside branch coverage, so the search explores thread schedules and not just inputs (Xu et al. 2020)2.
  • MUZZ — thread-aware gray-box fuzzing for multithreaded programs; it instruments the target to stress the thread-interleaving states ordinary coverage ignores (Chen et al. 2020)3.

These target the same interleaving bugs as systematic concurrency testing, but by coverage-guided sampling rather than systematic or PCT-bounded exploration.

Curated input corpora

  • Big List of Naughty Strings — a community-curated list of strings that have historically broken parsers and validators (Unicode edge cases, RTL marks, format-string syntax, SQL injection patterns, very long strings). A string-handling fuzz harness can seed from it, and so can a property-test generator that needs realistic edge cases.

When to use, when not

Use:

  • Any code that parses untrusted input (network protocols, file formats, deserialization, regex compilation).
  • Security-sensitive C/C++ code regardless of input source.
  • Open-source libraries that handle parsing or crypto — submit to OSS-Fuzz.
  • Linux kernel paths or syscall interfaces — syzkaller.

Don't:

  • As the only verification. Fuzzing finds crashes; logic bugs need an oracle the harness supplies (properties, differential references, invariants).
  • For pure business logic with no crash risk. The marginal value is low compared to property tests.
  • For UI code, where crashes are caught by the platform and the failure modes are visual.

Running a productive campaign

  • Semantic boundaries beat raw bytes. Blind bit-flips and truncation die at the first validity check and never reach interesting code. The bugs come from a plausible-but-wrong structure the target will follow: a corrupt length or offset field, a record count, a type tag — values the parser trusts. Structure-aware fuzzing encodes the format so the fuzzer mutates fields rather than bytes (Google n.d.)4, and FormatFuzzer derives such a fuzzer from a binary-format specification (Dutra et al. 2021)5.
  • A strict contract beats "the process stayed up." A parser that leaks a raw RangeError or NullPointerException to its caller keeps the process up, and so does one that hangs, so an oracle asking only whether the process survived passes both. Under a stricter contract every rejection carries the library's own typed error and no input hangs the run, so a raw runtime error or a timeout becomes a finding rather than a pass — which is what turns a crash fuzzer into a robustness fuzzer without a full differential oracle.
  • Bugs surface one layer at a time. Bug N+1 stays invisible until bug N stops throwing, so a campaign proceeds in rounds: fix a class, re-fuzz, move to the next layer. A binary reader peels in that order: a bounds error, then over-allocation, then a corrupt count driving a huge array, then a view over a short slice, each hidden behind the previous. Several rounds is the normal cost.
  • Seed diversity beats mutation volume. A handful of structurally different valid inputs reaches far more code than one input mutated repeatedly.
  • The oracle usually costs more than the RNG. When a campaign is slow the cost is almost always per-iteration work — building the input, or an expensive oracle — rather than input generation. An oracle that materializes a large result the property never inspects can dominate wall-clock, and bounding the inputs so the oracle stays cheap can cut runtime at the same coverage. Profiling the phases comes before parallelizing them.
  • A fast gate and a deep campaign are different jobs. A property-based generator (fast-check, Hypothesis) gives a structured oracle and shrinks a failure to a minimal counterexample, which makes it an always-on CI gate: a fixed seed, seconds a run. A coverage-guided byte fuzzer (libFuzzer, Jazzer/Jazzer.js, AFL++) steers into deep branches blind mutation cannot reach, which makes it a non-deterministic campaign against a corpus and a time budget. Neither subsumes the other.
  • The harness is the asset, not the fix list. A list of bugs fixed is a snapshot; a harness that stays in the repo keeps finding the next layer and guards against regressions, and its run count can rise on a schedule. Untrusted input crosses in at two doors, and each wants a harness: the read side (arbitrary bytes) and the write side (arbitrary input into the builder, then a build-then-read round-trip property).

Recurring bug classes in parsers and binary readers

  • Sequential reads off the end. A bound on the shared read primitive makes an out-of-bounds read throw the typed error rather than a raw RangeError. Wrapping each call site individually is what creates the gap: the bound belongs where bytes become values.
  • Allocation sized by an attacker-controlled length word. A corrupt length field drives an enormous allocation, and an in-range multi-GB request succeeds before it OOMs, so an "is it allocatable at all" guard misses it. A clamp to the known end of input at the chokepoint catches what trusting the word does not. A speculative or prefetch read clamps to a truncated result instead of throwing, so a fire-and-forget read cannot reject unhandled, and the awaited path surfaces the corruption typed.
  • Sentinel-terminated decode loops. A loop that scans "until the terminating bit or byte" runs forever past a truncated region, because the out-of-range read returns a value that never matches the sentinel: one input hangs the process, which is a denial of service. A counted length bounds such a loop where the sentinel alone does not, and in byte-stream code such loops hide behind for (;;) and while (true).
  • Lazy "ensure-then-read" windows. An accessor that assumes a prior ensure step ran must still bounds-check its index against the real count — corruption reaches the read without the ensure. Every such accessor needs a typed corruption guard.
  • Cross-structure invariants are reachable by corruption. When the set a read assumes was prepared comes from a different on-disk structure than its read key, corrupt bytes can desync the two, firing an internal "you must call X first" assertion (meant as a programming-bug signal) on data corruption. Those guards need the typed corruption error too, not a bare Error.
  • Every corrupt-input guard must throw the library's typed error. A guard that throws a bare Error reads as a crash to the harness and to the caller. One typed error type at the deserialize boundary fixes most of these classes at once.

Evidence

  • OSS-Fuzz public counters (May 2025): >13,000 vulnerability fixes, >50,000 bug fixes across >1,000 projects (Google 2025)6.
  • Kernel bug lifetimes. One analysis of 125,183 Linux kernel bug-fix pairs found that the share of bugs fixed within a year of being introduced rose from ~0% (2010) to ~69% (2022). Its author attributes the shift to better tooling, and names dedicated fuzzing infrastructure as the probable reason some subsystems' bugs are caught fastest (Qu 2026)7. The source is a blog analysis of kernel git history, not a peer-reviewed study, so the percentages are indicative.
  • Csmith found 325+ unknown compiler bugs in GCC, LLVM, and others over three years (Yang et al. 2011)8.
  • The CompCert outlier: Csmith found no wrong-code bugs in its formally verified middle-end (Leroy 2009; Yang et al. 2011)9 8 — the contrast that bounds what fuzzing alone reaches.

Exploring the program's behavior space

All explore many program behaviors; they differ in what they run and what they guarantee. Model checking works on a spec or model and is exhaustive over it. Symbolic execution runs the actual code, solving path constraints to reach branches a fuzzer can't, with a bounded-exhaustive verdict; bounded model checking runs the actual code too, but poses a single SMT query over the whole program unrolled to a fixed depth rather than exploring paths one at a time. Fuzzing runs the actual code too but samples inputs empirically: cheaper per case, no exhaustiveness claim.

Perturbation testing: what you perturb, and where

All three deliberately subject a system to bad conditions, differing in what they perturb and where. Fuzzing generates inputs (random, coverage-guided, or grammar-aware) and watches for crashes, undefined behavior, or sanitizer trips. Fault injection perturbs the environment in a test harness (a failed allocation, a dropped packet, a crashed dependency) to check the error-handling and fault-tolerance paths. Chaos engineering injects the same kind of environment faults into a running production system, verifying fallbacks under real conditions. Fuzzing probes input handling; fault injection and chaos probe failure resilience, in a harness and in production respectively.

Generative testing

These methods blur together because they all run the code and check the result without a hand-written expected value. Two roles pull them apart, and a single test picks one of each.

Input generators choose the input, and differ by the steering signal (the Generative subtree of the input axis):

Oracle suppliers provide the verdict when no expected value is written:

Pick one generator and one oracle: they compose. A coverage-guided fuzzer that checks a metamorphic relation is fuzzing and metamorphic at once. (The written-answer end, where the author picks rows and answers by hand, is example / parameterized tests.)

Classification

  • Quality dimensions: Security, Functionality, Reliability.
  • Area: Parsers, security-sensitive C/C++, kernels and syscall surfaces, network protocols, crypto, compression and image decoders.
  • Guarantee: Empirical.

Referenced by

References

  1. Rigger, Manuel, and Zhendong Su. 2020. "Testing Database Engines via Pivoted Query Synthesis." Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI '20). https://doi.org/10.48550/arXiv.2001.04174. ↩

  2. Xu, Meng, Sanidhya Kashyap, Hanqing Zhao, and Taesoo Kim. 2020. "Krace: Data Race Fuzzing for Kernel File Systems." 2020 IEEE Symposium on Security and Privacy (SP), 1643–60. https://doi.org/10.1109/SP40000.2020.00078. ↩

  3. Chen, Hongxu, Shengjian Guo, Yinxing Xue, et al. 2020. "MUZZ: Thread-aware Grey-box Fuzzing for Effective Bug Hunting in Multithreaded Programs." 29th USENIX Security Symposium (USENIX Security 20), 2325–42. https://doi.org/10.48550/arXiv.2007.15943. ↩

  4. Google. n.d. Structure-Aware Fuzzing with libFuzzer. Google/fuzzing. https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md. ↩

  5. Dutra, Rafael, Rahul Gopinath, and Andreas Zeller. 2021. FormatFuzzer: Effective Fuzzing of Binary File Formats. https://doi.org/10.48550/arXiv.2109.11277. ↩

  6. Google. 2025. OSS-Fuzz: Continuous Fuzzing for Open Source Software. Google/oss-fuzz, GitHub. https://github.com/google/oss-fuzz. ↩

  7. Qu, Jenny Guanni. 2026. Kernel Bugs Hide for 2 Years on Average. Some Hide for 20. Pebblebed Blog. https://pebblebed.com/blog/kernel-bugs. ↩

  8. Yang, Xuejun, Yang Chen, Eric Eide, and John Regehr. 2011. "Finding and Understanding Bugs in C Compilers." Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '11), 283–94. https://doi.org/10.1145/1993498.1993532. ↩↩

  9. Leroy, Xavier. 2009. "Formal Verification of a Realistic Compiler." Communications of the ACM 52 (7): 107–15. https://doi.org/10.1145/1538788.1538814. ↩