Opinion: AI Server Changes Need a Fault Drill, Not Just a Rollback Plan
A rollback plan tells you how to undo an AI change, but not what breaks first when the change stays in place. Most production incidents do not begin with a deliberate rollback; they begin with an unexpected failure mode that the author never tested. I now treat a passing fault drill as a precondition for reviewing any AI-generated server patch. The drill runs on a disposable server before a human…
Most production incidents do not begin with a deliberate rollback; they start with an unforeseen failure mode that the author never tested. A rollback plan informs you of how to restore the system to a known state, but a fault drill identifies what happens when the change encounters an unanticipated condition. A change might have a flawless rollback yet still fail in ways not noticed until the data is lost.
Consequently, a fault drill serves as essential evidence rather than just documentation. To conduct a fault drill, follow these five steps:
1. Generate failure modes before applying any changes. Utilize MonkeyCode's free model access to enumerate potential failure modes for the diff, avoiding any proposed fixes at this stage.
2. Provision a disposable server mirroring the target's OS and service manager, apply the change, and ensure the service starts smoothly. This server should be disposable as the next step entails deliberately inducing failures.
3. Execute the fault-drill.sh script against the unit and a sentinel file representing data the service must not lose. The script applies each fault individually, waits a few seconds, records the service state, attempts a manual restart, and checks the sentinel.
4. Review the classification table row by row, classifying each failure mode along three axes: service state post-fault, recovery after manual restart, and sentinel integrity.
5. Approve the change only if every tested failure mode is either boring or recoverable. Document the verdict next to the diff. If any fault results in lost data, the change fails review regardless of the diff's clarity.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
