Skip to content

Software Quality

Bugs in AI-generated code

AI-generated code carries a characteristic failure mode: output that compiles and reads as correct but does the wrong thing. As models write more of a codebase, that defect profile becomes the case verification has to catch.

What the defects look like

A 2025 systematic review of the defect literature sorts reported bugs into eight categories (functional, reliability, syntax, code style, hallucination, system, test, and unspecified) and finds functional bugs (the code does the wrong thing) the most common, discussed in 78% of the surveyed studies, followed by syntax bugs in 42% (Gao et al. 2025)1. Within functional bugs, the largest subcategories are semantic and logic bugs (26%) and bugs in variables, objects, parameters and properties (27%).

An execution study of code from nine LLMs, classifying each failure by running it against unit tests, also finds functional bugs the most common, with misunderstanding and logic errors the most prevalent subtype. It counts bugs rather than studies, and there syntax bugs are the least common of three types, not the second. On tasks taken from real GitHub repositories the share of runtime bugs, chiefly incorrect arguments and boundary checks, is significantly higher than on the standard benchmarks (Dou et al. 2024)2.

Semantic and logic bugs do not break compilation. Syntax bugs, by contrast, are the cheap case: a compiler rejects them at once.

Why it happens

The review attributes the pattern to two structural causes (Gao et al. 2025; Gao 2025)1 3:

  • The training corpus carries the defects. Models learn from public code, which itself contains bugs and uneven quality, so some of what they reproduce is wrong by inheritance. Strengths also track the corpus — a model is most reliable in the languages it saw most of.
  • Generation has no feedback. A model emits code without executing it or running a test against it, so nothing during generation catches a syntactically-correct-but-semantically-wrong result.

Bugs appear in every model family the review surveyed; larger models, and models better at following instructions, tend to generate fewer of them, not none.

Why it makes the oracle matter

A sound oracle in the loop (a compiler, a test suite, a sanitizer) supplies the feedback generation lacks, and that is what makes the consume relationship the strong one. The review reports the field moving the same way, from relying on static analysis alone toward "multi-layered intervention" combining program analysis, prompt engineering, and coding agents that generate, test, and debug code (Gao et al. 2025)1.

The same defect profile explains why the judge relationship is weak. Semantic and logic bugs are hard for an AI reviewer with no ground truth to catch: the code reads as correct, and only running it (or knowing the intent it was meant to satisfy) reveals otherwise.

Referenced by

References

  1. Gao, Ruofan, Amjed Tahir, Peng Liang, Teo Susnjak, and Foutse Khomh. 2025. A Survey of Bugs in AI-Generated Code. https://doi.org/10.48550/arXiv.2512.05239. ↩↩↩

  2. Dou, Shihan, Haoxiang Jia, Shenxi Wu, et al. 2024. "What's Wrong with Your Code Generated by Large Language Models? An Extensive Study." Science China Information Sciences. https://arxiv.org/pdf/2407.06153. ↩

  3. Gao, Ruofan. 2025. "Bugs in AI-Generated Code: Understanding Bug Patterns and Possible Fix Strategies." 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), 881–83. https://doi.org/10.1109/ICSME64153.2025.00098. ↩