I built an autonomous multi-agent AI pentester — and why it's not another GPT wrapper

The article discusses the development of a new AI pentester called OIHK, which stands out from other AI pentester projects in several ways. Instead of a single large language model running a while-loop to find vulnerabilities, OIHK utilizes a team of specialized agents to perform autonomous penetration testing. Each agent is responsible for a specific task, such as reconnaissance, discovery, validation, and reporting.

One key feature of OIHK is its use of a versioned scan plan, revision history, and an evidence ledger to track progress and maintain a clear audit trail. Agents operate independently, claiming steps in the plan, attaching real evidence, and updating state through a revisioned store. This ensures that a finding is only considered valid if there is real evidence and independent validation, rather than just a convincing prompt-generated report.

OIHK prioritizes safety by enforcing strict guardrails in its code rather than relying on prompts. The engine is designed with offline, sandboxed execution to prevent any accidental or malicious actions outside the defined scope. It is provider-agnostic, meaning it can work with any OpenAI-compatible endpoint, allowing users to switch models as needed without altering the underlying engine.

In addition to being an autonomous penetration testing tool, OIHK also functions as an evaluation environment for assessing AI models. It runs the engine against 16 locally hosted, intentionally vulnerable scenarios and programmatically scores the model's performance. This enables more reliable benchmarking and assessment of AI models in real-world scenarios.