175 Items Marked Done, Zero Delivered: An AI Agent Queue Incident

On August 30, 2026, a problem emerged within an AI agent queue system that had gone unnoticed for several days. The issue manifested despite the system logging a successful completion of its operations. This was not a quiet failure, but rather a success that yielded no results. The problem originated from the system's deduplication ledger, which erroneously marked 175 items as processed before they were actually handled by the processing pipeline.

Consequently, these items never made it to their intended downstream systems, leaving them completely unprocessed.

The core of the issue was the order in which operations were performed. When new data items arrived, they first passed through a deduplication ledger, which was supposed to prevent duplicate processing of the same item. The ledger would mark an item as "done" if it had already been processed. However, this marking happened before the system enforced a processing cap, a limit designed to manage resource usage and prevent overwhelming downstream services.

Items exceeding this cap for a given processing cycle were meant to be held back for the next cycle. This led to the critical flaw that these items were marked as done without actually being processed.

To address the issue, the solution involved reordering the operations. The processing cap needed to be applied before the deduplication ledger marked items as done. This ensured that only items that had genuinely passed through the processing cap and were either being processed or had completed processing were marked as done. This change was implemented in commit 36e985c of the VibeJobHunterAIPA_AIMCF on August 30, 2026, which directly addressed the problem of items being discarded due to the incorrect application of the processing cap and the deduplication ledger.

This incident highlighted a common pitfall in designing AI agent pipelines, where the interaction between different operational layers can lead to silent data loss if not carefully considered. It emphasized the need for robust monitoring that goes beyond simple success metrics. The system had reported success because the deduplication ledger was updated, but the actual output was zero, a discrepancy that required specific checks.

The monitoring setup was found to be lacking as it did not include a direct comparison of items ingested versus items delivered downstream, with an expected delta.

The incident also revealed a lack of shared context between AI agents and human operators, which made debugging more challenging. The operator queue, as documented, explicitly stated that no agents could see each other's chats or messages, relying solely on HubSpot and this file for shared information. Automating the detection of logical inconsistencies, such as the discrepancy between items ingested and items delivered, is now a priority.

In response to the incident, the FAQ section addressed several key points. It clarified that the 175 items were determined by the internal logs of the data source, which showed these items were successfully ingested and passed to the next stage. However, the downstream system showed zero, indicating the items were marked as done without processing.

It also noted that the high restart counts seen in other processes were unrelated to this specific issue, indicating different classes of problems. The incident served as a reminder to implement specific metrics for new data sources, such as comparing items ingested versus items delivered downstream, to avoid similar issues in the future.