Review Agent PRs for Silent Interface Drift

A checkout squad opened a pull request on Monday, claiming a small null-safety cleanup. The diff appeared minor in GitHub. The helper function loadInvoice ceased throwing NotFoundError, instead returning null. Three unit tests were updated to align with this change. External callers were not modified. However, staging transformed missing invoices into HTTP 500 errors.

This scenario serves as an illustrative example, not a post-mortem. The issue Agents commonly address is altering a function's contract. JavaScript remainders parse the contract unchanged; tests stay green as the agent concurrently revised them. Reviewers who merely skim diff hunks may overlook this. The problem lies at the export boundary, where return shapes, thrown errors, and field names drift.

The article introduces a "caller-inventory review." Each altered export is classified, and reviewers must either trust, revert, or test that change. To begin, one must examine the callers relying on the original export. These callers catch typed errors and map them to HTTP 404. The NotFoundError class extends Error, with a constructor that maps an invoice ID to a custom error message, sets the error name and status.

The loadInvoice function is exported as an async function, fetching invoice data from the database and returning it if found, or throwing a NotFoundError otherwise.

An agent PR may rewrite this file, presenting a cleaner implementation but incompatible with existing handlers. The new body checks for empty rows, returning null if no row is found, or a modified row with updated amount if present. Two contracts simultaneously changed - the thrown error type and the returned data shape. Handlers anticipating NotFoundError now encounter null.

Workers reading invoice.total now face undefined. Unit tests conceal both shifts since the agent also revised them. To perform a comprehensive review, follow these five steps on every agent PR, beginning with examining the frozen public surface, not focusing on style nitpicks. The first step involves freezing the old export surface - checking out the merge base, saving touched exports, and generating a diff.

This "freeze file" functions as the contract to be compared against the PR. Next, inventory remaining call sites - JavaScript won't list these automatically. A small script aids in identifying them. The script utilizes recursion to traverse directories, reading JavaScript files and searching for the altered export names.