Skip to content

Software Quality

AI tooling for tests

AI writes tests here, and the runner that executes them cannot tell whether they assert anything worth asserting. A survey of LLMs across software-testing tasks found test-case preparation and program repair the most-studied uses (Wang et al. 2024)1. The field moves quickly, and these entries may be out of date.

Example tests

  • Meta TestGen-LLM (Alshahwan et al. 2024)2 — deployed at Meta on Kotlin; 73% of its test improvements were accepted by developers and landed in production. Research-with-deployment, not a shipping tool.
  • TestPilot (Schäfer et al. 2023)3 — research; iterative repair to raise coverage on npm libraries.
  • CoverUp (Pizzorno and Berger 2025)4 — research; puts the uncovered lines and branches in the prompt, and asks again when coverage does not improve.
  • CODAMOSA (Lemieux et al. 2023)5 — research; Pynguin plus an LLM, to escape coverage plateaus.

Property-based testing

Hypothesis Ghostwriter is a pre-LLM symbolic generator. A PBT benchmark paper (Vikram et al. 2023)6 found LLMs write properties that run without error far more often than meaningful and correct ones. The realistic tool is a general-purpose LLM in the IDE.

Fuzzing

  • OSS-Fuzz-Gen (Google) — LLM-driven harness generation integrated with OSS-Fuzz. Google reports it found new vulnerabilities including a long-latent OpenSSL bug.
  • Fuzz4All (Xia et al. 2024)7 — research; LLM-based universal fuzzer.
  • TitanFuzz / FuzzGPT (Deng, Xia, Peng, et al. 2023; Deng, Xia, Yang, et al. 2023)8 9 — research; fuzzing for deep-learning libraries.
  • ChatAFL (Meng et al. 2024)10 — research; LLM infers state structure for stateful protocol fuzzing.
  • WhiteFox (Yang et al. 2024)11 — research; compiler-optimization fuzzing.

An LLM in the byte-mutation loop adds latency that usually loses on raw throughput; the value is at the harness / grammar level.

References

  1. Wang, Junjie, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. "Software Testing With Large Language Models: Survey, Landscape, and Vision." IEEE Transactions on Software Engineering 50 (4): 911–36. https://doi.org/10.1109/TSE.2024.3368208. ↩

  2. Alshahwan, Nadia, Jubin Chheda, Anastasia Finegenova, et al. 2024. "Automated Unit Test Improvement Using Large Language Models at Meta." Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE 2024), 185–96. https://doi.org/10.1145/3663529.3663839. ↩

  3. Schäfer, Max, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. https://doi.org/10.48550/arXiv.2302.06527. ↩

  4. Pizzorno, Juan Altmayer, and Emery D. Berger. 2025. "CoverUp: Effective High Coverage Test Generation for Python." Proceedings of the ACM on Software Engineering, 2897–919. https://doi.org/10.1145/3729398. ↩

  5. Lemieux, Caroline, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. "CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models." 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 919–31. https://doi.org/10.1109/ICSE48619.2023.00085. ↩

  6. Vikram, Vasudev, Caroline Lemieux, Joshua Sunshine, and Rohan Padhye. 2023. Can Large Language Models Write Good Property-Based Tests? https://doi.org/10.48550/arXiv.2307.04346. ↩

  7. Xia, Chunqiu Steven, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. "Fuzz4All: Universal Fuzzing with Large Language Models." Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 1–13. https://doi.org/10.1145/3597503.3639121. ↩

  8. Deng, Yinlin, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. "Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models." Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 423–35. https://doi.org/10.1145/3597926.3598067. ↩

  9. Deng, Yinlin, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023. Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT. https://doi.org/10.48550/arXiv.2304.02014. ↩

  10. Meng, Ruijie, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. "Large Language Model guided Protocol Fuzzing." Proceedings 2024 Network and Distributed System Security Symposium (NDSS). https://doi.org/10.14722/ndss.2024.24556. ↩

  11. Yang, Chenyuan, Yinlin Deng, Runyu Lu, et al. 2024. "WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models." Proceedings of the ACM on Programming Languages, 709–35. https://doi.org/10.1145/3689736. ↩