48-Hour Field Notes: My Output File Looked Complete Because I Streamed Straight Into It
A batch job that writes one JSON document per item sounds like the safest code you will write all week, right? No shared state, no transactions, no concurrent writers, just open() , write() , close() . So why did I spend most of two days chasing a downstream consumer that kept failing on files which were, by every check I had, present and readable? Here is the short version: my writer streamed…
The author discovered that a batch job writing JSON documents per item, streaming model output directly into the destination path, led to unexpected failures. The resume logic treated the existence of a file as completion, causing issues when the process died mid-stream. The writer concatenated chunks of structured text, creating a valid sequence of lines even if the file was cut at 60%.
The downstream consumer read four records instead of six and reported success. The author identified two distinct bugs that compounded each other. The first bug was truncated output that still parsed, while the second was a resume logic that skipped the damage. The first bug resulted from writing to the file directly, while the second was due to checking for the file's existence instead of validating its content.
The author shared a harness they created to reproduce the failures and fix the issues. They recommended writing to a temporary file with the .part extension, flushing it to disk, and then renaming it to the final destination. Additionally, they suggested validating the content of the temporary file before promoting it to the final location.
To ensure the fix works, they suggested running a crash test by killing the worker mid-stream and asserting that the destination file never appeared.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
