Skip to content

Software Quality

AI tooling for domain-specific verification

Domain-specific methods lean on expert judgment, so AI stands in for the expert rather than feeding a checker. The field moves quickly, and these entries may be out of date.

Threat modeling

STRIDE-GPT enumerates STRIDE threats from an architecture description, and Auspex (Crossman et al. 2025)1 (research) encodes threat-modeling tradecraft in the prompts that drive the same enumeration. PentestGPT (research) drives penetration testing against a running system rather than a design. The recurring failure is false comprehensiveness: a plausible threat list that misses the domain-specific threat, which keeps the output advisory rather than authoritative.

Supply-chain hygiene

Socket.dev (commercial) scores supply-chain risk with behavioral analysis of new package versions and has flagged real compromises in the wild. GitHub's Dependabot groups updates heuristically with an LLM description layer; Copilot Autofix extends to supply-chain findings. Independent evaluation is thin: the evidence is vendor-reported. The part that reduces real risk, reachability triage, is static analysis with an LLM front-end rather than an AI verdict, so the non-AI engine is doing the work.

Load and stress testing

AI tooling here is mostly test-script generation, the easy part. k6 Studio generates scripts from recorded traffic; Gatling Enterprise (commercial) summarizes a run; Locust is a non-AI baseline. None of them does the hard part, a realistic load profile.

Profiling

AI value is summarization, not analysis. The summary features sit in the commercial products, not in the open-source collectors under them: Grafana Cloud Profiles over Pyroscope, Polar Signals Cloud over Parca, Datadog in its own Continuous Profiler. Root-cause analysis stays with the non-AI profiler and a human.

References

  1. Crossman, Andrew, Andrew R. Plummer, Chandra Sekharudu, Deepak Warrier, and Mohammad Yekrangian. 2025. "Auspex: Building Threat Modeling Tradecraft into an Artificial Intelligence-based Copilot." 2025 IEEE Conference on Artificial Intelligence (CAI), 1160–67. https://doi.org/10.1109/CAI64502.2025.00201. ↩