Repository Agent-Security Gap Study — v2h baseline

A study has examined how different types of malicious content in repository files are detected by various scanning tools. The research focused on agent-directed manipulation in repository content and compared the effectiveness of an agent-facing detector (Sentinel InjectionGate) with current repository scanners such as secret detection and SAST (static application security testing).

The study used a baseline dataset consisting of 192 files, including 118 synthetic agent-directed payloads and 72 clean controls. The agent-facing detector, Sentinel InjectionGate, was able to flag all 118 agent-directed payloads, as well as 13 additional types of malicious content that are not considered agent-directed, bringing the total detection rate to 85%.

In contrast, the secret detection tool gitleaks correctly identified the single planted secret but missed the planted eval/0.0.0.0 file. The SAST tool semgrep detected one of the planted secrets and four code vulnerabilities, but again, missed the other agent-directed payloads.

The study also found that Sentinel InjectionGate was able to sanitize the prompt for 43 out of the 118 hostile files, effectively neutralizing the malicious code. However, 20 of the hostile files still reached the large language model (LLM) without being sanitized, indicating that there is still a risk of the malicious content being executed by the LLM.

Overall, the study highlights the limitations of current repository scanners in detecting agent-directed manipulation and demonstrates the effectiveness of Sentinel InjectionGate in identifying and mitigating these threats.