How to run AI security code review on GitHub pull requests without alert fatigue

To conduct AI security code reviews on GitHub pull requests without becoming overwhelmed by false alarms, follow a two-step process. First, employ deterministic rules to identify immediate red flags such as secrets, removed safety guards, and protected paths. These rules run automatically and categorize pull requests into auto (low risk), review, or block lanes. The AI step only comes into play if the pull request is placed in the review or block lane.

The deterministic rules are defined in a .mergejudge.yml file located in the base branch. These rules include coarse secret screening, removal of safety guards, and protection of certain paths. Any changes to .mergejudge.yml are not allowed within the pull request itself.

Optional security review mode can be enabled for teams that desire a more thorough security audit, particularly when reviewing AI-agent pull requests or changes to backend, webhooks, and authentication components. This mode, when activated, evaluates five high-severity criteria that focus on trust-boundary breaches and potential security vulnerabilities. These criteria include untrusted input reaching a sink, authorization checks, credentials and cryptography issues, inbound callbacks, and the use of AI tools.

The confirmed findings from the AI review, if confirmed by examining the actual changes in the pull request, automatically promote the pull request to the review lane. Findings that need further validation remain in the auto lane, pending human review to ensure accuracy and prevent false alarms. This structured approach ensures that AI security reviews are both efficient and reliable, maintaining a balance between thoroughness and mitigating alert fatigue.