Cheap LLM code review is fine until it hits an authorization bug

A code review service tested two large language model (LLM) options—a $1.20-per-million-output-token model called Luna and a more advanced one called Astra—against 50 public code samples from popular open-source projects. The results showed a stark difference in bug detection between the two systems. Luna identified 69 bugs for a cost of $0.20, while Astra found 92 bugs for $5.66, making Luna 20 times cheaper per verified bug.

However, when it came to security bugs, Astra outperformed Luna, catching 19 out of 24 security issues compared to Luna's 9. The most concerning discrepancy was in the Keycloak project, an authentication and permission server. Astra found 14 security bugs, while Luna only discovered 6, many of which were false positives. The two Keycloak bugs caught by Astra but missed by Luna were particularly problematic: recovery codes could be reused indefinitely, and a new global view permission silently ignored individual client permissions.

This demonstrates that LLM models have blind spots when it comes to high-risk authentication and permission code. While Luna could serve as a triage layer for routine changes, it is crucial to route any code involving authentication, authorization, or permission logic to a more reliable model for review. The cost difference between the two models is overshadowed by the potential damage of a missed security bug.