Skip to content

Software Quality

How AI fits into software quality

AI changes the economics of verification, but not uniformly. What sorts the effect by method is AI's relationship to the oracle: whether AI is the oracle, consumes one, or writes one. That is the split this section is organized on — the oracle being whatever decides an output is correct (oracle axis). The same shift also makes AI a prolific source of the code being checked, with a characteristic failure mode — plausible but subtly wrong — surveyed in bugs in AI-generated code. A companion page on agent experience takes the supply side: how a codebase's readability and feedback loops set what an agent can do in it, and why that can be measured, unlike human developer experience. An agent that holds tools is also a system to be verified rather than a tool to verify with, and that turns it back into an ordinary security target. Agent security testing is the method, classified in the domain family alongside penetration testing.

That relationship comes in three kinds. Where no fast, sound oracle exists, AI's verdict has to stand in for one — AI is the oracle. Where one exists, AI either produces the thing being checked (it consumes the oracle) or authors the check itself (it writes the oracle). The division is an organizing convenience, not a theory; it earns its place because each kind predicts how far to trust AI. A method can sit in more than one — example tests are written by AI (an authored oracle) and run under a test runner (a consumed one).

A dated tool catalog runs one page per method family, mirroring the methods catalog. A snapshot, not a directory.

AI is the oracle

No sound oracle exists, so a model's verdict stands in for one: LLM as judge, a reward model, an AI code review. The only ground truth is human consensus, which the verdict approximates. The one correctness lever is the scaffold (a checklist, STRIDE, a rubric) — completeness, since soundness is unavailable.

Current state

The densest tooling and the thinnest evidence. Code review, requirements and threat-modeling copilots, and the observability / AIOps summary layer cluster here; the tooling is advisory and vendor claims dominate. AI fails predictably at intent-vs-implementation drift, production semantics, and social context.

Judging intent and design

Method What AI does Trap
Threat modeling STRIDE enumeration from an architecture description (STRIDE-GPT, Auspex (Crossman et al. 2025)1) False comprehensiveness — a plausible list that misses the domain-specific threat
Code review PR review (Open Code Review, PR-Agent); catches low-level defects (null derefs, missing error handling, leaked secrets); design-level review is attempted (Rasheed et al. 2024)2 but unproven, echoing human review's known skew to low-level issues (Bacchelli and Bird 2013)3 Treating AI review as a replacement rather than a triage pass
ADRs Drafts Context / Consequences prose; flags PRs touching a decision An ADR with no human decision behind it is decoration
Requirements quality checking Per-requirement ISO 29148 scoring (Lubos et al. 2024)4; ambiguity detection Cross-requirement contradiction detection is research-grade only
Refactoring practice Mechanical refactors at scale; the change runs under the test suite (a Tests oracle) What to refactor, whether it's worth it, and the social context have no sound oracle
Checklists Generates a launch / on-call checklist from a service description A generic list misses the routinely-skipped items that are the whole point

Judging runtime signals

Method What AI does Trap
Monitoring and observability Triage: cluster correlated alerts, summarize log spikes (Notaro et al. 2021)5 SLO/alert catalogs from code alone are production-naive
Chaos engineering Experiment hypotheses from manifest inspection (ChaosEater (Kikuta et al. 2025)6, research) Designing a useful experiment needs production semantics (blast radius, tiering)
Change-point detection Reads the metric series and attributes the regression to a diff Pinning environmental drift (hardware, input changes) on a commit
Supply-chain hygiene CVE-noise triage, SBOM diffs; Socket.dev behavioral risk Reachability triage is static analysis with an LLM front-end, not an AI verdict
Load and stress testing Script generation from an API spec (k6 / Locust / Gatling) A realistic load profile is production-semantics work AI doesn't do
Profiling Flame-graph triage; correlates profiles with deploy timestamps Root-causing is contextual (cache topology, allocator, GC) — hint-generator, not analyst
Microbenchmarking Interprets benchmark results —
Canary / shadow launches Synthesizes canary-metric deltas into a verdict, drafts the rollback Promotion decisions stay statistical (Kayenta since 2018)
A/B experimentation Drafts hypotheses, summarizes results, flags imbalances Doesn't change the statistics (variance reduction, sequential testing, sample-ratio mismatch)
Exploratory testing Vision-LLM browser agents notice broken UI; convert a found bug into a Playwright repro Lacks the charter sense and domain curiosity a human brings

Trap

Plausible output nobody can cheaply check, so the verdict stays advisory. Shared evaluation is what separates a real signal from vendor claims: Code Review Bench scores review tools on precision and recall against human-verified pull requests, with a refreshed online set so a tool can't memorize the cases.

AI consumes an oracle

A compiler, test runner, sanitizer, or solver already says yes-or-no; AI proposes code or inputs and the oracle decides — an external oracle in the loop.

Current state

The only consistently strong relationship. The oracle gives feedback on the code AI writes, so it can iterate toward something correct rather than merely plausible, and every empirically successful AI tool in the survey works this way. Sound by construction: the oracle rejects wrong output however plausible the proposal looked.

Method Oracle in the loop Trap / note
Linear types rustc (DARPA TRACTOR) The clearest consume relationship — AI cannot ship an unsafe program through rustc; C→Rust of non-trivial modules at scale is still unreliable
Type systems the compiler (on shape) Types that compile but are wrong about intent — an any slipped past the checker, an over-narrow union
Effect systems the effect checker Thin training data (Koka/Eff/Unison); realistic use is annotating mainstream code with what it does
Refinement and dependent types an SMT solver AI proposes predicates and repairs failed verifications — genuine new capability (Dafny + LLM)
Static analysis CodeQL / Semgrep; autofix (Copilot Autofix, Project Naptime) Finds a different distribution (semantic / intent bugs) at higher false-positive rates
Clone detection a clone detector (jscpd, PMD CPD, Simian) No AI-specific tool needed — a general agent consumes the report and proposes the de-duplication
Dead-code detection a detector (knip, Vulture) + tests Confirming a removal is safe needs a second oracle — the test suite plus whole-program reachability
Example tests the test runner, via compile-run-repair AI tests written from AI-written code share the blind spot — write them from an independent spec (Siddiq et al. 2024)7
Property-based testing the property Authoring the property is the weak link (see AI writes an oracle)
Fuzzing sanitizer + harness (OSS-Fuzz-Gen, Fuzz4All (Xia et al. 2024)8) AI solves the harness / seed / grammar first-mile; it doesn't run the engine
Differential testing an independent reference implementation AI can't supply the oracle — inventing one shares the blind spot; its role stays proposer-side
Symbolic execution a KLEE / angr harness AI drives the harness and prioritizes path-explosion output; it does not replace the engine
Theorem proving the proof kernel (Lean Copilot (Song et al. 2024)9) Genuinely new: automates most proof steps for short lemmas (human-in-the-loop, not yet autonomous; see AI writes an oracle)
Decision tables, state machines, executable specifications, model checking, statistical model checking a model checker / rule engine AI drafts the artifact; fidelity to intent is the failure mode (see AI writes an oracle)
Statistical testing, deterministic simulation testing the test / the simulator AI authors the workloads and assertions; the test or simulator adjudicates

Trap

Gaming the oracle, or the oracle being weaker than assumed — a proposal that satisfies the check without satisfying intent: a test suite passed by overfitting to its cases, a program that type-checks only because the hard parts were widened away.

AI writes an oracle

AI authors the check itself: a property, a contract, a schema, a spec, a snapshot. The historical blocker was the cost of writing the check, and that is what AI absorbs.

Current state

Weak, consistently, across independent studies. The recurring shape is syntactically valid far more often than meaningful: weak property quality (Vikram et al. 2023)10, type-correct-but-semantically-wrong annotations (Yee and Guha 2023)11, mediocre single-shot tests (Siddiq et al. 2024)7, and weak conformance of generated TLA+ models to the systems they claim to model (Cheng et al. 2025)12. AI does lower the authoring cost, which is why methods once rejected as too expensive to write are worth recalculating; but the recalculation has to price in the validation cost the authoring cost shed. The one genuinely new capability is narrow: proving short lemmas and repairing failed verifications in theorem proving and refinement types (DeepMind 2024)13, where the solver still adjudicates.

Authoring a check from intent

Method Check AI authors Trap
Type systems TypeT5 (Wei et al. 2023)14 Inferred types compile yet miss intent (Yee and Guha 2023)11 — encode existing assumptions, not intended types
Refinement and dependent types Laurel (Mugnier et al. 2025)15, Clover (Sun et al. 2023)16 An authored predicate may encode existing behavior, not the intent
Property-based testing AI proposes properties and generators (Vikram et al. 2023)10 Skew too weak (tautologies) or too strong
Contracts and runtime assertions LLM ranks Daikon-mined invariants (Ernst et al. 2007)17; intent→postcondition (Endres et al. 2024)18 Inferring invariants for a non-trivial module encodes existing assumptions, not intended contracts
Metamorphic testing Matches the relation catalog (identity, idempotence, round-trip, algebraic laws) to a function No dedicated tool; a general agent writes the relations
State machines Stately Studio AI Fidelity to intent — admits transitions the designer didn't mean
Executable specifications Specula (spec synthesis) Same; the artifact is small enough to audit (the method's cost floor anyway)
Decision tables the table and its test cases Compiles but admits cells the designer didn't mean
Model checking Drafts TLA+ models (Cheng et al. 2025)12 Syntactically valid, poor conformance (SysMoBench)
Theorem proving AlphaProof (DeepMind 2024)13 — short lemmas Long proofs and novel definitions stay human
Deterministic simulation testing Workload generators, assertions over simulated state The simulator is the work; AI triages failed seeds and summarizes counterexamples

A second pattern — the artifact constraining the model — appears at the lightweight tier (decision tables, state machines, executable specifications); see the formal-methods AI page. No tool or study yet covers AI authoring effect-system annotations (niche languages, untested) or statistical testing / statistical model checking designs — a general agent drafts the check and a human validates it against intent.

Authoring a check from observation

Method Check AI authors Trap
Snapshot / approval tests the scaffold Cannot approve — and approval is the whole operation; an LLM rubber-stamping the first run is the sharpest authored-oracle trap
Characterization tests (legacy code) Turns observed behavior into a guard before a refactor (the Feathers move) Freezes current behavior, bugs included
Mutation testing Codebase-specific mutation operators; killing tests for survivors A signal, not an oracle (see Signals); triage survivors into equivalent / write-an-assertion / real-gap

Trap

A solver or proof kernel keeps the code honest against the authored check, but nothing keeps the check honest against intent: the authored oracle passes something other than what was meant. Sharpest in a characterization test that bakes current (maybe buggy) behavior in as "expected."

Signals

The signal methods — coverage, git-history hotspots, mutation testing — are things AI reads, not oracles; AI uses them to target where to write the next test or mutant. An AI reads a high-churn file and proposes what's structurally wrong — the causes, not rankings finding (Lewis et al. 2013)19; it generates the test for an uncovered line (the CoverUp / Qodo Cover loop); it pipes Code Maat hotspot output through an assistant.

Referenced by

References

  1. Crossman, Andrew, Andrew R. Plummer, Chandra Sekharudu, Deepak Warrier, and Mohammad Yekrangian. 2025. "Auspex: Building Threat Modeling Tradecraft into an Artificial Intelligence-based Copilot." 2025 IEEE Conference on Artificial Intelligence (CAI), 1160–67. https://doi.org/10.1109/CAI64502.2025.00201. ↩

  2. Rasheed, Zeeshan, Malik Abdul Sami, Muhammad Waseem, et al. 2024. AI-powered Code Review with LLMs: Early Results. https://doi.org/10.48550/arXiv.2404.18496. ↩

  3. Bacchelli, Alberto, and Christian Bird. 2013. "Expectations, Outcomes, and Challenges of Modern Code Review." Proceedings of the 35th International Conference on Software Engineering (ICSE '13), 712–21. https://doi.org/10.1109/ICSE.2013.6606617. ↩

  4. Lubos, Sebastian, Alexander Felfernig, Thi Ngoc Trang Tran, et al. 2024. "Leveraging LLMs for the Quality Assurance of Software Requirements." Proceedings of the 32nd IEEE International Requirements Engineering Conference (RE '24), 389–97. https://doi.org/10.1109/RE59067.2024.00046. ↩

  5. Notaro, Paolo, Jorge Cardoso, and Michael Gerndt. 2021. "A Survey of AIOps Methods for Failure Management." ACM Transactions on Intelligent Systems and Technology, 1–45. https://doi.org/10.1145/3483424. ↩

  6. Kikuta, Daisuke, Hiroki Ikeuchi, and Kengo Tajiri. 2025. "LLM-Powered Fully Automated Chaos Engineering: Towards Enabling Anyone to Build Resilient Software Systems at Low Cost." 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 3861–65. https://doi.org/10.1109/ase63991.2025.00331. ↩

  7. Siddiq, Mohammed Latif, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. 2024. "Using Large Language Models to Generate JUnit Tests: An Empirical Study." Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE 2024), 313–22. https://doi.org/10.1145/3661167.3661216. ↩↩

  8. Xia, Chunqiu Steven, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. "Fuzz4All: Universal Fuzzing with Large Language Models." Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 1–13. https://doi.org/10.1145/3597503.3639121. ↩

  9. Song, Peiyang, Kaiyu Yang, and Anima Anandkumar. 2024. Lean Copilot: Large Language Models as Copilots for Theorem Proving in Lean. https://doi.org/10.48550/arXiv.2404.12534. ↩

  10. Vikram, Vasudev, Caroline Lemieux, Joshua Sunshine, and Rohan Padhye. 2023. Can Large Language Models Write Good Property-Based Tests? https://doi.org/10.48550/arXiv.2307.04346. ↩↩

  11. Yee, Ming-Ho, and Arjun Guha. 2023. "Do Machine Learning Models Produce TypeScript Types That Type Check?" Proceedings of the 37th European Conference on Object-Oriented Programming (ECOOP 2023). https://doi.org/10.48550/arXiv.2302.12163. ↩↩

  12. Cheng, Qian, Ruize Tang, Emilie Ma, et al. 2025. SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems. https://doi.org/10.48550/arXiv.2509.23130. ↩↩

  13. DeepMind. 2024. AI Achieves Silver-Medal Standard Solving International Mathematical Olympiad Problems. https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/. ↩↩

  14. Wei, Jiayi, Greg Durrett, and Isil Dillig. 2023. TypeT5: Seq2seq Type Inference using Static Analysis. https://doi.org/10.48550/arXiv.2303.09564. ↩

  15. Mugnier, Eric, Emmanuel Anaya Gonzalez, Ranjit Jhala, Nadia Polikarpova, and Yuanyuan Zhou. 2025. "Laurel: Unblocking Automated Verification with Large Language Models." Proceedings of the ACM on Programming Languages, 1519–45. https://doi.org/10.1145/3720499. ↩

  16. Sun, Chuyue, Ying Sheng, Oded Padon, and Clark Barrett. 2023. Clover: Closed-Loop Verifiable Code Generation. https://doi.org/10.48550/arXiv.2310.17807. ↩

  17. Ernst, Michael D., Jeff H. Perkins, Philip J. Guo, et al. 2007. "The Daikon system for dynamic detection of likely invariants." Science of Computer Programming 69 (1–3): 35–45. https://doi.org/10.1016/j.scico.2007.01.015. ↩

  18. Endres, Madeline, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. "Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?" Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE 2024), 1889–912. https://doi.org/10.1145/3660791. ↩

  19. Lewis, Chris, Zhongpeng Lin, Caitlin Sadowski, Xiaoyan Zhu, Rong Ou, and E. James Whitehead. 2013. "Does Bug Prediction Support Human Developers? Findings from a Google Case Study." Proceedings of the 35th International Conference on Software Engineering (ICSE). https://doi.org/10.1109/icse.2013.6606583. ↩