AI Is Not a Source of Truth

AI systems are not reliable sources of truth, according to recent discussions. While coding agents can be adept at identifying suspicious code within a repository, their findings are not definitive. After an agent highlights a potential issue, the burden of proof falls on the human reviewer. It's important to remember that the agent may have overlooked a guard, misunderstood code calls, misinterpreted configuration, or generated an explanation that doesn't hold up to scrutiny.

Agreement from another AI model does not equate to proof; it's merely additional confirmation. To truly validate a claim, one must run the code, reproduce the behavior, write a failing test, examine the actual configuration, and consider other relevant factors such as log files, dependency versions, and Git history. Even when the claim is partially verified or found to require unconfirmable assumptions, acknowledging "Couldn't verify this" is valuable information.

The author argues that static analysis and coding agents should not be considered competing approaches. Static tools excel when there's a clear understanding of what needs to be checked, while coding agents can uncover situations where no rule has been written yet. As coding agents become more capable of generating and inspecting code, the distinction between their findings and definitive proof becomes increasingly crucial.

It is essential to treat an agent's identification of a bug as a question that needs to be rigorously proven rather than accepting the model's confidence as the final answer.