{
  "id": 7085070,
  "title": "AI Security Scanning Needs Evidence, Not Just More Agents",
  "url": "https://urgent.news/2026/09/13/ai-security-scanning-needs-evidence-not-just-more-agents",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-13T08:47:20.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sudeephazra/ai-security-scanning-needs-evidence-not-just-more-agents-2mlb"
  },
  "original_language": "en",
  "account": "Google's Mantis demonstrates a key challenge in AI security scanning: identifying vulnerabilities doesn't equate to proving they actually exist. This nuance is often overlooked in most AI security demonstrations. Security teams are accustomed to noisy scanners, misleading alerts, and findings that demand human interpretation to distinguish real exploitation paths from theoretical concerns. While an LLM can enhance a scanner's fluency, it could also produce more sophisticated inaccuracies - a less desirable outcome. The standout feature of Mantis is not the \"agentic vulnerability scanning\" label, a buzzword trending in software development. Instead, the value lies in the grounding structure it employs: repository context, history, threat models, reviewer stages, critic stages, sandboxed reproduction, and patches linked to evidence. This structure is logical because vulnerability detection is a multifaceted task, not a singular job. The reproduction step stands out as the most crucial. If a system can produce a working crash, a failing test, an exploit path, or a concrete data-flow issue, the review process transforms. Security teams no longer rely solely on model confidence; they examine tangible evidence. However, this doesn't eliminate human judgment. Instead, it equips humans with useful information to judge. Most teams struggle not due to a shortage of alerts, but because alerts lack sufficient context. Someone must open the repository, grasp the service boundary, trace input handling, inspect validation, examine framework behavior, review past fixes, evaluate the path's reachability, and debate the scanner's output - a labor-intensive process. Machine learning can assist with this work, but only if it's designed as a workflow rather than an all-powerful tool. For instance, a scanner stating \"Possible SQL injection in UserController.java\" falls short. A more effective system should provide context like: This route accepts user input here. The value reaches this query builder here. This sanitizer doesn't cover this pattern. Here's a minimal reproducer. Here's the failing test. Here's the proposed fix. And here's the residual uncertainty. The final point is crucial. A security tool that claims every finding is equally certain sets bad incentives. Engineers may start to disregard it, security teams might reduce its sensitivity, and the tool will eventually become background noise. The promise of an agentic scanner is to break down work into stages, allowing each stage to review and challenge the previous one. One agent may identify suspicious flows, another may critique the finding, another may attempt reproduction, and another may propose a patch. The final stage could verify if the patch impacts behavior beyond the intended scope. This mirrors a careful human review process. It also reshapes my perspective on model selection. Not every stage requires the most powerful model. Tasks like classification, deduplication, clustering similar findings, and summarizing repository structure differ from tracing a subtle authorization bypass through multiple application layers. A pragmatic system should allocate its reasoning budget where it's truly needed—a sensible SRE approach. The model isn't magic; it's just one part of a workflow that includes cost, latency, permissions, state, logs, and failure modes. The cost aspect is easy to overlook in a research announcement but can be costly in a real-world engineering environment. If every pull request triggers a deep multi-agent investigation across a large monorepo, the expenses could become significant. The system requires triage. It needs fast filters, expensive analysis only when justified, and clear rules for synchronous CI checks versus asynchronous security pipeline checks. Timing is also an important factor. Some checks should occur directly in the developer loop. They must run swiftly, fail clearly, and deliver results while the developer remembers their changes. Other checks are better suited for background analysis. A comprehensive investigation across service boundaries might be valuable, but it shouldn't halt every commit unless the organization is ready to bear the operational cost. This highlights the need to separate developer feedback from security investigation. Developer feedback demands speed and clarity. Security investigation demands depth and evidence. Attempting to merge these into a single workflow usually results in a system that's too sluggish for developers and too superficial for security teams - a common pitfall observed in static analysis, dependency scanning, data-quality checks, and policy-as-code. AI doesn't eliminate this trade-off; instead, it makes it harder to notice initially. I'd propose structuring the workflow like this: This approach keeps the resource-intensive reasoning stages near the findings that truly merit them. While the diagram appears clean, there are still substantial operational tasks hidden behind it. The scanner must incorporate sandboxing, restricted network access, deterministic handoffs between stages, audit logs, and a mechanism to prevent aggressive false-positive filters from dismissing credible findings. It also requires ownership, as once a tool begins filing security bugs or proposing patches, someone must determine what constitutes \"good enough.\" This is where security automation can become contentious. A tool that merely reports findings is easily ignored. A tool that generates patches is more noticeable but also riskier. The patch might resolve the immediate issue while altering behavior elsewhere. It might suppress a test instead of addressing the root cause. It might introduce a new vulnerability.",
  "summary": "Google’s Mantis caught my attention because it points to a problem most AI security demos quietly walk around: finding a vulnerability is not the same as proving one exists. That distinction matters. Security teams already live with noisy scanners, half-useful alerts, and findings that require someone experienced to separate a real exploit path from a theoretical complaint. Adding an LLM can…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}