That Time a Rolling Deployment Almost Corrupted Our Data

A critical data pipeline, recently migrated from Python to Rust, experienced a near-miss data corruption issue during a deployment. The pipeline produces batches of files every 2.5 minutes, which downstream systems rely on for automated decision-making. The ingestion path does not verify the correctness or completeness of the files, so if two different processes produce inconsistent outputs, downstream decisions become unreliable.

During the migration, the team utilized the orchestrator's rolling deployment feature, which allows for a smooth transition between tasks. However, in this case, the rolling deployment inadvertently caused parallel execution of both the old and new tasks. This resulted in both tasks generating files concurrently and writing to the same storage paths without coordination.

The bug was discovered during beta testing when the new task and the old task concurrently wrote files to the same paths. Task B completed its run, writing the first half of the files, then received a termination signal and stopped. Task A continued writing the second half of the files. Due to the unstable sorting algorithm used by the Rust dataframe library, identical inputs could produce records in different orders across separate runs.

This resulted in the first half of the dataset coming from Task B and the second half from Task A, with the records in incompatible sequences.

When the ingestion process consumed the dataset, the ordering assumptions it relied on no longer held, leading to corrupted data. The only indication of the issue was overlapping timestamps that should never have existed, and no alerts or monitoring alerts were triggered. It took two engineers manually inspecting logs to identify the problem.

To prevent such issues in the future, the team explored two possible fixes. Option 1 involved using a distributed lock in a key-value store. Before writing files, a task would acquire the lock. If another task already held the lock, the new task would wait. During the deployment window, only two processes (the old and new tasks) would contend for the lock, ensuring a clean handoff.

Option 2 involved generating a unique identifier for each task and writing the output to a directory named after that identifier. Overlapping runs would then produce output in separate locations, preventing any potential conflicts.

The team chose Option 2, as it provided isolation without relying on an external system. While both options addressed the issue, Option 1 offered a cleaner design that explicitly defined the failure boundary. However, the team's decision to go with Option 2 demonstrated a trade-off between consistency and availability risks. After implementing the fix, overlapping runs and mismatched dataset timestamps became rare, ensuring more predictable and reliable deployments.