The bug every AI coding tool ships, and how to prove it is gone

An open-source scanner named auditai-scanner, available at github.com/audit0/auditai-scanner, has been designed to identify a specific bug in AI coding tools. This bug allows unauthorized access to sensitive data, such as viewing another user's invoice, by manipulating the URL. The scanner's rules are transparent and can be reviewed before trusting any findings.

To demonstrate the bug, the source code of an invoicing app was analyzed, showing how the service-role key bypasses row-level security, enabling unauthorized access. The scanner's output provides minimal information, necessitating additional manual checks by developers. However, the scanner sets a higher standard for fixing bugs, requiring reproduction using a sandbox environment and a regression test.

The fix implemented in the invoicing app ensures that the request runs as the signed-in user, allowing the row-level security policy to determine access. The regression test verifies this behavior by sending the request as a different user (Alice) and expecting a 403 Forbidden response. The scanner's results are accompanied by a coverage line, indicating the number of routes checked, verified, confirmed, and unverified.

Unverified findings are not considered failures but indicate that the scanner could not reproduce those specific issues. The scanner is designed to be run locally, without requiring any account or sending data to external servers. To utilize the scanner, developers can install it using npx auditai-scan. If you have a Next.js + Supabase app and would like a real authorization bug identified and proven for free, there are ten design-partner slots available at auditai.sh.