Postmortem: Preemption Split a Multi-File Agent Apply

A failed agent job can leave a repository in an unfinished state, with tests passing on an incomplete, unreviewed tree. The solution is atomic staging combined with a checksum manifest. This postmortem details a controlled lab reproduction of the issue. On shared CI runners, agent jobs writing multiple files risk losing the process when preempted.

MonkeyCode provides free resources for testing, but the lesson applies broadly. Teams need a durable, crash-safe write protocol to handle sudden process death. In the lab, a Python service with four modules was used, each responsible for a specific task. The intended patch required renaming a required field across all modules. When the apply process was interrupted after only two file writes, the remaining files still reflected the old contract.

The unit tests only covered the files that were successfully written, giving a false impression of correctness. The reproduction used a tiny Python service with four modules, including one for request schema, validation, storage, and HTTP handling. The issue lies in the apply process writing files one at a time without any lease or heartbeat mechanism.

Without proper coordination, a premature exit leaves the repository in a broken state. The reproduction did not use real traffic, but the same pattern occurs on preemptible free servers. To prevent this, an atomic apply protocol must be implemented. The protocol includes staging files in a designated directory, creating a manifest with file details and hashes, sending heartbeat signals every few seconds, swapping files only when the manifest is complete, and rolling back if the process fails mid-swap.

If the apply process is interrupted at any point, the original tree remains intact. Implementing this protocol ensures that the repository is always in a consistent state, even in the face of unexpected interruptions.