178 reports in one afternoon: what a publish burst does to an LLM pipeline

On a Thursday afternoon, a traveler published 178 trip reports to his Polarsteps diary, starting at each stop. He had made nine reports before this. Each report, when published, initiates three background jobs: text moderation, place extraction, and image moderation for any photos. All these jobs call a model, running them in the background, not simultaneously. The queue that processes these jobs is what this report examines.

After the publish, the worker claimed ten oldest jobs per minute, each with a five-minute lease. It was the pace that kept within the model's per-minute budget. Six minutes after the publish, the queue showed 152 moderate-content jobs pending, 152 moderate-image jobs pending, and 0 image moderation jobs done. This processing is the hidden cost of every publish.

The queue contained four jobs that were still pending, each under a live lease. There was also a cron job calling the worker every minute, timing out after five seconds. This was the cron giving up, while the worker kept running.

A hundred photos were marked as needing review due to image moderation service being unavailable. These were caused by the model failing three times to moderate each photo. The model's refusal was due to the image moderation function failing its calls to the model.

The fix was to implement a queue system, allowing 10 reports to be processed per minute without failure. The second fix involved migrating photos with pipeline failures back to pending, so they are re-queued and processed by the worker. This prevented the model's failure from being mistaken for a verdict. The lesson learned was about separating failures from verdicts in the database.