Celery visibility_timeout: 214 LLM Jobs Ran Twice on Redis
The bug showed up as a spreadsheet problem. My nightly pipeline summarizes long PDFs with Claude and writes one row per document. One morning, document 4471 had two summaries. Slightly different wording, start times 61 minutes apart, same Celery task ID. Same task ID is the part that made my stomach drop. That isn't a retry. That's the broker handing the exact same message to a second worker…
Celery visibility_timeout determines how long the Redis transport waits for an acknowledgement before assuming a worker has failed and reinserting the message into the queue. Due to this timeout, Redis has no way of knowing whether a worker has simply run out of time on a long-running job or has crashed. Over 14 nights, Celery's default visibility_timeout of 3600 seconds caused 214 out of 1380 jobs to be run twice unnecessarily, as Redis repeatedly delivered the same message to different workers.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.