I Fact-Checked Every Finding an AI Security Reviewer Gave Me. Here's What Held Up.

During my summer job at RakFort, I was given the task of testing every security check in Secfoo using the Claude AI agent, and verifying whether the AI's reports were accurate. This post details my findings and offers advice for those who are new to using AI for security reviews.

For those unfamiliar, Secfoo is an open-source tool that integrates AI coding agents (such as Claude Code, Codex, Gemini CLI, Cursor, and GitHub Copilot) into a security review process. After identifying a check, you provide clear instructions and a standardized report format, which results in a structured report containing findings, severity levels, and recommendations. The tool also offers a local dashboard for easy access to the reports.

In my testing, Secfoo's own codebase served as my target, as it provided a comprehensive test since I could examine every file the AI referred to. For each report, I performed the following steps:

1. Opened every finding in the report.

2. Visited the exact file and line number the AI pointed to.

3. Verified the code at the exact commit date of the scan, rather than using today's code.

4. Classified the finding as Confirmed, Partly Wrong, or Wrong.

5. Compared the numbers on the dashboard with the report.

It was more crucial than initially thought to ensure that the AI's findings were accurate. Codebase changes occur daily, so checking an old report against the current code would falsely accuse the AI of errors.

Here are the results of my tests:

- Deployment readiness: Out of 7 findings, 6 were completely correct.

- Severity: F1 was High, F2 was High (partly wrong), F3 was Medium, F4 was Medium, F5 was Medium, and F6 and F7 were Low.

- Responsible AI compliance: All four findings were accurate, and the low overall rating made sense for a tool that doesn't make decisions about people.

One notable finding (F2) was that while the CI system didn't run any security scanners, and the linter was set to never fail the build, the report also incorrectly stated that the release workflow ran on any v* tag push. The issue stemmed from a comment in the workflow file explaining the change, which was made five days before the scan. The AI had incorporated this outdated information into the report, highlighting the importance of checking the correct version of the code.

I also tested the following aspects:

- Bad input: Supplying incorrect folder names, file names, unknown skill or agent names, and nonexistent GitHub repos all yielded clear error messages without crashing or wasting AI tokens.

- Dashboard: All pages loaded correctly, and finding counts matched the reports except for the home page, which crashed with a 500 error due to a recent change. A simple one-line fix resolved the issue, and my PR (#31) was merged.

Key lessons learned:

1. AI reviewers are genuinely useful, as most findings were accurate with correct file and line numbers. Their coverage and speed are significant benefits for small teams.

2. Professional-looking reports don't guarantee accuracy. The one incorrect detail was presented with the same confidence as the correct findings.

3. Always check against the correct version of the code. Use the commit from the exact scan date to avoid falsely marking accurate findings as incorrect (or vice versa).

4. Human oversight is crucial for findings that matter. Even though AI reviewers save time, every finding should still be verified before being presented to someone who will sign off on it.

Secfoo is free and open source (MIT license). If you're a student or developer interested in security, I highly recommend testing Secfoo on your own project and verifying the findings yourself. This is the best way to understand the strengths and weaknesses of AI reviewers. You can find Secfoo at https://github.com/secfoo-com/secfoo. Feedback, issues, and contributions are welcome. I look forward to hearing about your experiences using Secfoo!