AI Workflows Need a Dry-Run Contract

Developers often assume AI automation will run smoothly, but in reality, the true challenge lies in managing the side effects of these workflows. An AI agent can quickly read a ticket, inspect a repository, call an API, and open a pull request, but the question arises: can we predict the consequences before they occur? The answer isn't simply to add more logs, as logs only reflect what happened after the workflow ran.

Instead, we need a preview that outlines the workflow's intended actions, the permissions required, and the reversible actions. This concept is referred to as a "dry-run contract," which establishes a clear agreement between the AI workflow, the tools it uses, and the person approving the results.

The core issue is distinguishing between low-risk actions, such as reading files and proposing a patch, and high-risk actions, like sending a message, deleting a test fixture, rotating credentials, or changing a production setting. These high-risk actions involve crossing the boundary from reasoning into the real world. To address this, it is crucial to make side effects explicit by listing each proposed action with a stable ID, its target, and its expected consequence. Additionally, the preview should indicate when an action cannot be reversed.

To ensure clarity, the dry-run contract proposes using three distinct states: Preview, Apply, and Verify. The Preview phase collects intended actions without making any changes to external state. The Apply phase executes only the actions that have been approved. Finally, the Verify phase checks the observed results and reports any discrepancies from the preview. These phases should all use the same action format, even if they employ different implementations.

One key aspect of the dry-run contract is the introduction of a three-state workflow: Preview, Apply, and Verify. Unlike previous boolean-based approaches, this model avoids the risk of forgetting a branch that carries out the real action. By having the workflow in three visible states—Preview, Apply, and Verify—developers can clearly understand the workflow's intended actions, the approved actions, and the observed results, ensuring that any failures can be traced back to a concrete point.