AI Review Loops Don't Always Stabilise

The idea of AI reviewing code and implementing fixes in a loop is appealing, but it doesn't always result in a flawless outcome. Some developers inadvertently create a slow version of this loop where an AI-generated pull request (PR) is reviewed by one developer, then AI-fixed by another, and the process repeats indefinitely. However, this approach can create significant issues if not carefully managed with proper guardrails.

The primary problem lies in the inconsistency of AI opinions. The AI's understanding of what constitutes good code can vary from one run to another. In the worst-case scenario, the AI may even flip-flop between different opinions in successive reviews, leading to a never-ending loop. This inconsistency can make it difficult to achieve stable results.

Moreover, AI reviews can inadvertently encourage scope creep. For instance, a review might suggest adding a missing test case, and then it could escalate to recommending the implementation of a CI/CD pipeline. The list doesn't stop there; it can extend to demanding an Android app for seemingly unrelated features. This kind of scope creep can lead to new defects and complications in the development process.

Additionally, false positives or hallucinations in AI reviews can introduce new defects into the codebase. A quick test was conducted by prompting an AI model, Opus 5, to generate a small amount of 'perfect, non-trivial' code. The generated code was then passed through three review-fix loops. The results were alarming; the defect count increased each time the code went through the review-fix cycle. This underscores the risks associated with relying solely on AI for code reviews and fixes.