My AI-written plan went through every check I could think of. The gaps were in what I never said.
The plan was 22,000 words long. Twelve agents had worked on it for close to two hours. Among them, three architects had drafted it, two judges had scored the drafts, a synthesis had merged them, and three reviewers running on a different model had tried to break the result. They raised 51 findings. The plan accepted all 51. Its claims about our code came with file and line references. I read it…
The AI-generated plan consisted of 22,000 words and was developed by twelve agents working for nearly two hours. Three architects drafted the plan, two judges scored it, and three reviewers using a different model attempted to identify flaws. A total of 51 findings were raised, but all were accepted by the final version of the plan. The plan contained detailed file and line references for its claims about the code.
The writer, who manages engineering at GoodBarber (an app platform), commissioned the plan for a significant system change. He did not disclose the exact system or the plan's content. The writer aimed to identify where the AI process failed and what changes he would make.
Claude Code was used to create the plan, with the regular model interviewing the writer, describing a neighboring service, and establishing rules for the subsequent agents. The costly, more capable model was chosen for the planning phase, while the regular model handled executing milestones and reviewing the plan. The regular model was instructed not to review its own work.
The brief, about 2,500 words, was created by the regular model to serve as a reference for the agents' work. The process involved multiple iterations, with three architects each generating a plan from a different perspective. Two judges evaluated the plans for security and feasibility, resulting in a total of 51 findings. After revisions, the final plan consisted of 13 milestones and 32 open questions, each accompanied by the costly model's reasoning.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
