Skip to content

Software Quality

AI tooling for runtime and production

Runtime methods have little AI-native tooling: what exists is mostly a feature inside a general-purpose assistant, or a classical technique that never needed a model. The field moves quickly, and these entries may be out of date.

Contracts and runtime assertions

No mature AI-native contract tool exists; contract generation is a feature inside general-purpose assistants, which write Pydantic or dataclass validators from docstrings. Daikon (Ernst et al. 2007)1 mines invariants classically and is the non-AI baseline for separate research that translates natural-language intent into formal postconditions (Endres et al. 2024)2. That work stops at the research stage: nothing yet turns inferred invariants into enforced runtime guards, the step that would carry the technique from documentation into a live check. Contracts and runtime assertions covers the method itself, written contracts as well as inferred ones.

Monitoring and observability

Query authoring is the capability that holds up: Honeycomb's Query Assistant solves a query problem, not a causal-inference one. The major observability platforms (Datadog, New Relic, Grafana Sift, Dynatrace, Splunk, PagerDuty) add a real summarization layer; their root-cause claims are overstated. AIOps is the research area behind those features (Notaro et al. 2021)3.

Chaos engineering

Production platforms (AWS Fault Injection Service, Chaos Mesh, LitmusChaos, Steadybit) have no AI features worth naming. ChaosEater (Kikuta et al. 2025)4 is a research LLM agent that designs and runs experiments; it is not production-ready.

References

  1. Ernst, Michael D., Jeff H. Perkins, Philip J. Guo, et al. 2007. "The Daikon system for dynamic detection of likely invariants." Science of Computer Programming 69 (1–3): 35–45. https://doi.org/10.1016/j.scico.2007.01.015. ↩

  2. Endres, Madeline, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. "Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?" Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE 2024), 1889–912. https://doi.org/10.1145/3660791. ↩

  3. Notaro, Paolo, Jorge Cardoso, and Michael Gerndt. 2021. "A Survey of AIOps Methods for Failure Management." ACM Transactions on Intelligent Systems and Technology, 1–45. https://doi.org/10.1145/3483424. ↩

  4. Kikuta, Daisuke, Hiroki Ikeuchi, and Kengo Tajiri. 2025. "LLM-Powered Fully Automated Chaos Engineering: Towards Enabling Anyone to Build Resilient Software Systems at Low Cost." 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 3861–65. https://doi.org/10.1109/ase63991.2025.00331. ↩