Reviewing AI-generated code at volume: the ceilings per forge

AI code review systems typically assess quality before deploying them to handle large volumes of pull requests. The primary issue arises when an agent opens numerous requests daily, each requiring multiple iterations before completion. The three critical ceilings that cause problems are the forge’s API quota, the vendor’s review quota, and the reviewer’s trigger cadence. These three components are part of different products, owned by separate teams, and fail in various ways that resemble unrelated issues.

The forge’s API quota manifests as an apparent outage, while the vendor's quota appears as a reviewer going quiet. The reviewer’s cadence looks like a reviewer who stops noticing new commits. Each ceiling has set numbers, and for self-managed forges, these numbers are configurable. Forge quota comes first since all other components share it.

An AI reviewer functions as an API client, consuming database, compute, and storage resources with every call, sharing the same quota as other pipeline, script, and dashboard activities.

Azure DevOps provides the most transparent numbers, with throughput units (TSTU) measuring consumption. One TSTU represents a typical user's load over five minutes, with rate limits at 10 TSTU or less for normal activity and up to 100 for larger but less frequent spikes. The global limit is 200 TSTU within a sliding five-minute window.

If usage exceeds this limit, the response code is 429, which resembles a successful request but includes a Retry-After header. This deception leads to recording unexecuted requests, causing confusion between a review that found nothing and one that never occurred.

To avoid this issue, it is crucial to monitor X-RateLimit-Remaining and consider a non-zero delay as a failure. For Bitbucket Data Center, rate limiting is a token bucket per user, defaulting to 60 tokens with a refill rate of 5 tokens per second. It applies to HTTP requests using basic or bearer authentication but not to Git SSH operations.

When a bot misbehaves, two key considerations are providing a specific user an exemption or using a cluster with separate buckets for each user. GitLab's forge-side limits are documented per endpoint, primarily focusing on write endpoints like notes and comments.

The vendor ceiling refers to reviews per hour or credits per month, with CodeRabbit and GitLab offering different metering systems. CodeRabbit meters reviews, with per-developer, per-hour allowances ranging from 1 to 12, depending on the plan. Files per review also have caps, and exceeding them results in failure. GitLab uses credits, with a monthly commitment pool and on-demand usage at $1 per credit.

The billing method varies based on the setup, with self-hosted instances billed as a flat fee under an Enterprise License Agreement.