I Stopped Reviewing Code And Started Reviewing Agents

One incident at work revealed a critical issue with AI-powered agents. The agent had made a code change that passed through CI and appeared reasonable in the PR. However, the team had never properly verified what the agent could do, what tools it could run, and where its authority ended. This incident completely changed the reporter's perspective on the review process.

Initially, the reporter didn't pay attention to what the agent was doing behind the scenes. If a problem arose while checking the code, they would report it as a bug and ask for it to be fixed. After the agent fixed the bug, the reporter would test it again and be satisfied with the result. However, they never noticed what the agent was actually doing.

For most teams, code review simply means one thing - someone reviews the code changes. One person makes the changes, another reviews them, and the review is the control. However, this is no longer true on many teams, and most teams have not openly acknowledged this shift. The pull request opens a few hundred changed lines across a dozen files, has a tidy description, and passes green CI.

The reviewer opens it, reads the description, spot-checks the migration and tests, and approves. No one reads the few hundred lines of code.

The reviewer's role has shifted from reading the code to reviewing the agent's behavior. The diff-based review relied on the assumption that writing code takes longer than reading it. One person produces a couple of hundred lines in a day, while a colleague can read them carefully in twenty minutes. This ratio held for decades, and it's why reading the diff worked as a control.

However, agents produce a day's worth of diff in under an hour, and a team can run several of them at once simultaneously. Reading has not gotten faster, so the ratio has flipped. The control no longer holds on its own.

When a team can't inspect every output, it starts inspecting the thing that generates the outputs - the agents. The agent's permissions file, system prompt, and tool list are now the artifacts whose contents decide what ends up in production. These files deserve a thorough review, just like any other code. The reporter suggests three questions to ask before trusting a PR from an agent: what can it call, what might it affect? These questions cover most of what matters in reviewing the agent.