AI tooling for domain-specific verification¶

Domain-specific methods rest on expert judgment, and their AI tools produce work that an expert or a non-AI tool still checks.

Threat modeling¶

STRIDE-GPT enumerates STRIDE threats from an architecture description, and Auspex (Crossman et al. 2025)¹ (research) encodes threat-modeling tradecraft in the prompts that drive the same enumeration. The recurring failure is false comprehensiveness: a plausible threat list that misses the domain-specific threat, which keeps the output advisory rather than authoritative.

Penetration testing¶

PentestGPT (research) drives penetration testing against a running system rather than a design.

Supply-chain hygiene¶

Socket.dev (commercial) scores supply-chain risk with behavioral analysis of new package versions and has flagged real compromises in the wild. A GitHub Dependabot alert can be assigned to an AI coding agent (Copilot, Claude or Codex), which opens a draft pull request with a proposed fix. Independent evaluation is thin: the evidence is vendor-reported. The part that reduces real risk, reachability triage, is static analysis with an LLM front-end rather than an AI verdict, so the non-AI engine is doing the work.

Load and stress testing¶

AI tooling here is mostly test-script generation, the easy part. k6 Studio generates scripts from recorded traffic and uses an LLM to find the session tokens and other dynamic values a replay must extract; Gatling Enterprise (commercial) summarizes a run; Locust is a non-AI baseline. None of the AI features does the hard part: building a representative load profile, the request mix and volume that make a test resemble production traffic.

Profiling¶

The AI features sit in the commercial products, not in the open-source collectors under them. Grafana Cloud Profiles has an LLM explain a Pyroscope flame graph: its bottlenecks, their likely causes and suggested fixes. Datadog's Continuous Profiler summarizes its automated findings, and Polar Signals Cloud over Parca serves profiles to an external AI assistant through an MCP server. Each one interprets a profile the non-AI profiler collected.

References¶

Crossman, Andrew, Andrew R. Plummer, Chandra Sekharudu, Deepak Warrier, and Mohammad Yekrangian. 2025. "Auspex: Building Threat Modeling Tradecraft into an Artificial Intelligence-based Copilot." 2025 IEEE Conference on Artificial Intelligence (CAI), 1160–67. https://doi.org/10.1109/CAI64502.2025.00201. ↩