Skip to content

Software Quality

Bugs in AI-generated code

AI-generated code carries a characteristic failure mode: output that compiles and reads as correct but does the wrong thing. As models write more of a codebase, that defect profile becomes the case verification has to catch. A 2025 systematic review of the defect literature (Gao et al. 2025; Gao 2025)1 2 supplies the taxonomy and the distribution. It is one recent study, not yet independently replicated at scale, and the structural account is the more settled part.

What the defects look like

The review sorts reported bugs into eight categories (functional, reliability, syntax, code style, hallucination, system, test, and unspecified) and finds two dominate: functional bugs (the code does the wrong thing), in 78% of the surveyed studies, and syntax bugs, in 42% (Gao et al. 2025)1. Within functional bugs, the largest subcategories are semantic and logic bugs (26%) and bugs in variables, objects, parameters and properties (27%).

Semantic and logic bugs do not break compilation — the program runs and returns an answer; the answer is wrong. Syntax errors, by contrast, are the cheap case: a compiler rejects them at once.

Why it happens

The review attributes the pattern to two structural causes rather than a passing model weakness (Gao et al. 2025; Gao 2025)1 2:

  • The training corpus carries the defects. Models learn from public code, which itself contains bugs and uneven quality, so some of what they reproduce is wrong by inheritance. Strengths also track the corpus — a model is most reliable in the languages it saw most of.
  • Generation has no feedback. A model emits code without executing it or running a test against it, so nothing during generation catches a syntactically-correct-but-semantically-wrong result.

Bugs appear in every model family the review surveyed; larger, better instruction-following models shift the distribution without removing it.

Why it makes the oracle matter

A sound oracle in the loop (a compiler, a test suite, a sanitizer) supplies the feedback generation lacks, and that is what makes the consume relationship the strong one. The review reports the field moving the same way, from relying on static analysis alone toward "multi-layered intervention" combining program analysis, prompt engineering, and coding agents that generate, test, and debug code (Gao et al. 2025)1.

The same defect profile explains why the judge relationship is the weak one. Semantic and logic bugs are what an AI reviewer with no ground truth is worst at: the code reads as correct, and only running it (or knowing the intent it was meant to satisfy) reveals otherwise.

Referenced by

References

  1. Gao, Ruofan, Amjed Tahir, Peng Liang, Teo Susnjak, and Foutse Khomh. 2025. A Survey of Bugs in AI-Generated Code. https://doi.org/10.48550/arXiv.2512.05239. ↩↩↩↩

  2. Gao, Ruofan. 2025. "Bugs in AI-Generated Code: Understanding Bug Patterns and Possible Fix Strategies." 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), 881–83. https://doi.org/10.1109/ICSME64153.2025.00098. ↩↩