Security scanning at Foundation scale

In August 2026, the Apache Software Foundation (ASF) conducted comprehensive security scans across 230 of its repositories. This operation was carried out by the ASF Security and ASF Tooling teams, as part of the Foundation's Responsible AI Initiative (RAI), utilizing Anthropic's Claude Mythos 5 through their Project Glasswing security program.

The motivation behind these scans stemmed from two significant events. Firstly, the ASF Security team received specific Mythos-based scans from Alpha-Omega, which they analyzed and found valuable for their work. Secondly, there was an increasing influx of AI-generated vulnerability reports in open source projects, with many of these reports being unhelpful or misleading due to their lack of context and detail. This situation was expected to worsen regardless of any actions taken by the Foundation.

In response to these challenges, the ASF Security team developed a threat-modeling approach to ensure the generated security analysis was both relevant and worth reading. Meanwhile, ASF Tooling had been working on an automated audit pipeline to evaluate code against the OWASP Application Security Verification Standard (ASVS). Originally designed for code quality on Apache Trusted Releases, the pipeline proved general enough to be applied across various codebases within the Foundation, eventually becoming a managed service available to any ASF project.

The security scanning pipeline consists of three model tiers: a light tier for high-volume filtering, a medium tier for inventory building, and a heavy tier for in-depth analysis that requires actual reasoning. These tiers can be configured at runtime, allowing for a balance between quality, speed, and cost based on specific use cases. The pipeline operates on an ensemble of models, with flexibility to switch between different model combinations without affecting the pipeline itself.

A crucial aspect of the scanning process was providing the Large Language Models (LLMs) with contextual information about the codebase they were reviewing. This included details about security posture, architectural decisions, team code standards, and even private information about deployment choices. Incorporating this contextual guidance significantly reduced false positives and eliminated incorrect inferences from the final reports.

To facilitate the scanning process, ASF Security invited 75 Project Management Committees (PMCs) with over 180 repositories to participate in the threat model preparation. Each project described its security posture, detailing components that were fully trusted, those delegated to software operators, and what was explicitly out of scope.

A Threat Model skill contributed by Alpha-Omega assisted the Security Team in creating customized threat models, maintaining ongoing communications, and ensuring the models remained accurate and relevant to the codebase. The preparation resulted in scanning against reviewed threat models being roughly 20% more efficient compared to unreviewed scans, as the analysis focused on the most consequential parts of the system.