# AI tooling for tests

AI writes tests here, and the runner that executes them cannot tell whether
they assert anything worth asserting. A survey of LLMs across software-testing
tasks found test-case preparation and program repair the most-studied uses
(Wang et al. 2024)[^wang2024]. The field moves quickly, and these entries may be out of date.

## Example tests

- **Meta TestGen-LLM** (Alshahwan et al. 2024)[^alshahwan2024] — deployed at Meta on
  Kotlin; 73% of its test improvements were accepted by
  developers and landed in production. Research-with-deployment,
  not a shipping tool.
- **TestPilot** (Schäfer et al. 2023)[^schafer2023] — research; iterative repair to
  raise coverage on npm libraries.
- **CoverUp** (Pizzorno and Berger 2025)[^pizzorno2025] — research; puts the uncovered
  lines and branches in the prompt, and asks again when coverage
  does not improve.
- **CODAMOSA** (Lemieux et al. 2023)[^lemieux2023] — research; [Pynguin](https://github.com/se2p/pynguin)
  plus an LLM, to escape coverage plateaus.

## Property-based testing

[Hypothesis](https://hypothesis.readthedocs.io/) Ghostwriter is a pre-LLM symbolic
generator. A PBT benchmark paper (Vikram et al. 2023)[^vikram2023] found
LLMs write properties that run without error far more often than
*meaningful and correct* ones. The realistic tool is a
general-purpose LLM in the IDE.

## Fuzzing

- **[OSS-Fuzz-Gen](https://github.com/google/oss-fuzz-gen)** (Google) — LLM-driven harness generation
  integrated with [OSS-Fuzz](https://google.github.io/oss-fuzz/). Google reports it
  [found new vulnerabilities including a long-latent OpenSSL bug](https://security.googleblog.com/2024/11/leveling-up-fuzzing-finding-more.html).
- **Fuzz4All** (Xia et al. 2024)[^xia2024] — research; LLM-based universal fuzzer.
- **TitanFuzz / FuzzGPT** (Deng, Xia, Peng, et al. 2023; Deng, Xia, Yang, et al. 2023)[^deng2023a] [^deng2023] — research;
  fuzzing for deep-learning libraries.
- **ChatAFL** (Meng et al. 2024)[^meng2024] — research; LLM infers state structure
  for stateful protocol fuzzing.
- **WhiteFox** (Yang et al. 2024)[^yang2024] — research; compiler-optimization fuzzing.

An LLM in the byte-mutation loop adds latency that usually loses
on raw throughput; the value is at the harness / grammar level.

## References

[^wang2024]: Wang, Junjie, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. "[Software Testing With Large Language Models: Survey, Landscape, and Vision](https://arxiv.org/pdf/2307.07221)." *IEEE Transactions on Software Engineering* 50 (4): 911–36. <https://doi.org/10.1109/TSE.2024.3368208>.
[^alshahwan2024]: Alshahwan, Nadia, Jubin Chheda, Anastasia Finegenova, et al. 2024. "[Automated Unit Test Improvement Using Large Language Models at Meta](https://arxiv.org/pdf/2402.09171)." *Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE 2024)*, 185–96. <https://doi.org/10.1145/3663529.3663839>.
[^schafer2023]: Schäfer, Max, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. *[An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://arxiv.org/pdf/2302.06527)*. <https://doi.org/10.48550/arXiv.2302.06527>.
[^pizzorno2025]: Pizzorno, Juan Altmayer, and Emery D. Berger. 2025. "[CoverUp: Effective High Coverage Test Generation for Python](https://arxiv.org/pdf/2403.16218)." *Proceedings of the ACM on Software Engineering*, 2897–919. <https://doi.org/10.1145/3729398>.
[^lemieux2023]: Lemieux, Caroline, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. "[CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models](https://www.carolemieux.com/codamosa_icse23.pdf)." *2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)*, 919–31. <https://doi.org/10.1109/ICSE48619.2023.00085>.
[^vikram2023]: Vikram, Vasudev, Caroline Lemieux, Joshua Sunshine, and Rohan Padhye. 2023. *[Can Large Language Models Write Good Property-Based Tests?](https://arxiv.org/pdf/2307.04346)* <https://doi.org/10.48550/arXiv.2307.04346>.
[^xia2024]: Xia, Chunqiu Steven, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. "[Fuzz4All: Universal Fuzzing with Large Language Models](https://arxiv.org/pdf/2308.04748)." *Proceedings of the IEEE/ACM 46th International Conference on Software Engineering*, 1–13. <https://doi.org/10.1145/3597503.3639121>.
[^deng2023a]: Deng, Yinlin, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. "[Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models](https://arxiv.org/pdf/2212.14834)." *Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis*, 423–35. <https://doi.org/10.1145/3597926.3598067>.
[^deng2023]: Deng, Yinlin, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023. *[Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT](https://arxiv.org/pdf/2304.02014)*. <https://doi.org/10.48550/arXiv.2304.02014>.
[^meng2024]: Meng, Ruijie, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. "[Large Language Model guided Protocol Fuzzing](https://www.ndss-symposium.org/wp-content/uploads/2024-556-paper.pdf)." *Proceedings 2024 Network and Distributed System Security Symposium (NDSS)*. <https://doi.org/10.14722/ndss.2024.24556>.
[^yang2024]: Yang, Chenyuan, Yinlin Deng, Runyu Lu, et al. 2024. "[WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models](https://arxiv.org/pdf/2310.15991)." *Proceedings of the ACM on Programming Languages*, 709–35. <https://doi.org/10.1145/3689736>.

## Acronyms

- OSS — open-source software
- PBT — property-based testing