The Deploy That Broke Running Jobs: Changing an n8n Workflow While Executions Are In Flight

When running LLM workflows in n8n in production, changing a workflow while executions are in flight can be challenging. The key rule is to snapshot the current version before making any changes. This involves exporting the current version with a date and a one-line note about its function. Rollback becomes a simple reactivation of the old export, rather than a reconstruction effort.

Before deploying, it's crucial to drain the system. This means stopping new work from entering the workflow to avoid mixed old and new runs. Pause the trigger or deactivate the workflow to allow all in-flight executions to finish. Check the execution list for any jobs stuck in places like the dead-letter queue or review queue. Only after confirming zero in-flight executions should you then change the workflow.

After deploying, run a canary job through the new version first. This job should be a full end-to-end job with harmless payload. The canary helps catch issues like renamed fields or steps that only matter with a full payload. If the canary's output matches the expected result, only then re-enable the trigger. If the canary fails, rollback using the snapshot from earlier. Remember, rollback only restores the workflow definition and does not undo the side effects produced by the new version.

A rollback plan should include reactivating the old version and listing the jobs that ran on the new version so their outputs can be checked manually. This ensures that side effects from the new version are not double-applied when the workflow is rolled back. This process, while requiring extra steps, has saved the author from the pitfalls of mixed-version runs, making workflow deploys safer and more predictable.