I Let AI Plan 170 Changes. It Made the Same 3 Mistakes Every Time.

I ran 170 change-planning goals across 40 domains, including identity management, multi-agent operations, SRE, supply chain policy, and FinOps. Using PlannerCritic, an LLM writes a plan, a second LLM reviews it, and deterministic gates decide whether it can move forward. The cost was $0.49. The result was not that the model is bad, but rather a structural failure in planning, with three recurring mistakes: unverified dependencies, unsafe sequencing, and weak rollback.

These defects were not random or occasional, but rather three families of problems. Unverified dependencies occurred 57 times, with tasks declaring something must be true without ensuring it is true beforehand. Unsafe sequencing happened 46 times, with steps ordered before their prerequisites. Weak rollback happened 18 times, with high-blast-radius steps having rollback mechanisms that don't actually undo anything.

The 3 main defects were unverified dependencies, unsafe sequencing, and weak rollback. A bigger model didn't help, indicating it was a planning-structure problem, not a parameters problem. A revision loop was designed to converge, but the planner tended to fix one blocker and introduce another, failing to close dependency gaps or enforce ordering.

A higher-leverage change was a precondition closer, a deterministic linter that verifies every precondition maps to an earlier task. This eliminated 48% of the blockers without asking the model to get smarter.