Before I let an agent open PRs on strangers' repos, I made it earn it

The author, after merging 44 bug fixes into other open-source projects, realized that delegating such tasks to an agent could be risky. The author then developed a set of rules for the agent to follow before it can open pull requests (PRs) on strangers' repositories. The first rule, replay against what maintainers actually merged, involves running the agent on the code from just before each merged fix and comparing its patch with what the maintainer merged.

If the match is under ~60% or if the PR isn't something the author would send, the agent publishes an evaluation write-up instead of shipping. The second rule, a forward test on unseen repositories, requires the agent to run the full loop on repositories it has never seen before, save the drafts locally, and not post anything. The author found that only 2% of the 160 active Python MCP repositories had the bug they were testing for, making it rarer in the wild than initially thought.

The third rule, Fixture filter and size routing, involves reading the README and file paths to drop test data, benchmarks, examples, and vendored code. The agent also drafts an issue for larger changes to give the maintainer the option to choose the size of the change. The final rule is that a human sends every PR, so the agent only handles the slow parts of the process, such as reproducing the bug, patching it, testing it, and writing up the changes.

The author then tested the agent on five real repositories and found that four drafts were clean, which they sent after reviewing. However, the fifth repository, a trading framework, was too complex for the agent to handle, as it found that a fix for the bug would break working calls to the official MCP Python SDK. The agent also noticed that the scanner had missed a whole file of tools, which was fixed and documented in the source.