AI Image Generation Pipeline: 5 Queue Tests for 429 Object Storage Upload Retry Backoff
Short answer: Put generated report images in private object storage, but make generation and upload separate queue states; persist the generated bytes before upload, retry a 429 with bounded backoff, and derive the object key from the job ID so every retry targets the same object. For a logistics system serving generated reports to authenticated customers, retention is part of correctness. The…
The AI image generation pipeline should handle 429 object storage upload retry using a five-step test procedure. Generate the report images and store them in private object storage, but separate the generation and upload processes into different queue states. Persist the generated bytes before uploading, and use exponential backoff with jitter for retries when encountering a 429 status code.
Derive the object key from the job ID, such as report-job-id.png, to ensure the same key is targeted for each retry attempt. The upload worker reads the saved payload without calling the generator, and the key should remain deterministic, not including an attempt number or timestamp.
Key points to consider:
1. Model the pipeline as two independent retryable operations: generation and upload.
2. Persist generated bytes before uploading and use a deterministic object key.
3. Implement exponential backoff with jitter and a fixed attempt ceiling for retries.
4. Ensure byte reuse, stable identity, bounded delay, and single-writer coordination.
5. Verify deletion of the report after the declared retention period.
A Go program is provided as an example of an upload worker that uploads previously saved bytes, with an explicit route, method, and key verification. The code includes six attempts and a 30-second timeout as local policy inputs.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.