When Your Content Bot Hits an LLM Quota, Ship the Fallback

When a publishing bot based on a single LLM provider encounters an operational snag, the result can be a rather uninspiring failure mode. In one particular instance, the workflow appeared green, but the article module responded with "no action" following a generation failure with the "LLM unavailable" message. This type of failure may seem innocuous in continuous integration (CI) but can prove costly in a content pipeline.

The conventional approach to automating content creation and dissemination tends to merge the processes of generating content and publishing it. This convenience soon becomes problematic, however, once the generation stage fails after the scheduler, secret management, and publishing client have already completed their tasks. To mitigate such issues, it is advisable to decouple the generation process from the delivery process.

The publishing client ought not to concern itself with the origin of the article, be it an LLM, a template, or a manually reviewed draft. It should simply receive a standardized article object, irrespective of the generation method employed.

A fallback article should not attempt to masquerade as a freshly generated piece, complete with recent benchmarks, citations, or provider-specific pricing. Instead, it should succinctly encapsulate the operational lesson that has transpired. This approach ensures that users are not misled into believing that the fallback content possesses attributes it does not.

Key takeaways from this experience include the necessity of treating article generation and article publishing as separate failure domains. When LLM generation encounters a failure, the system should promptly return a fallback article, rather than an empty action list. It is also crucial to maintain the honesty of the fallback content by refraining from inventing benchmarks, pricing, or citations.

Moreover, any original error type should be recorded to prevent a successful publish from concealing underlying provider issues.

While this fallback article serves as a temporary solution, the ultimate objective is to implement a more robust system. This includes the adoption of a multi-LLM provider system capable of automatic switching in the event of quota exhaustion; the introduction of a quota monitoring dashboard to track usage across various providers; and the creation of a content buffer that stores pre-generated articles for emergency scenarios.