I Built a Multi-Provider LLM Router for My AI Worker - Here's What I Learned

When the author began integrating large language models (LLMs) into their side project AuraFlow AI, a common mistake made by many backend engineers occurred: they hardcoded a single LLM provider. This resulted in a significant setback when the Gemini free tier's rate limit was hit while testing, causing the system to stop functioning and forcing the author to manually swap the API key and restart the worker.

This experience taught the author the importance of avoiding a single point of failure when using LLM providers. AuraFlow AI is a distributed data cleaning system built for the author's portfolio, which consists of two main components: a POST /jobs endpoint (NestJS/Fastify/Bun) that pushes tasks to BullMQ, and Python LangGraph workers that pick up jobs, parse raw/malformed data using LLMs, validate outputs, and persist results to PostgreSQL.

The core of both agents is the LLM, making the system highly dependent on its availability. The author's initial implementation used the ChatGoogleGenerativeAI class from the LangChain Google Generative AI library with a hardcoded Gemini API key. However, this approach proved fragile as it failed to account for rate limits, system outages, and the need to switch to cheaper providers or test local models.

To address these issues, the author designed the LLMRouter, a registry-based provider abstraction with automatic fallback capabilities. The LLMRouter class maintains a chain of provider instances loaded based on a prioritized order specified in environment variables. When invoking the LLM, the router sequentially tries each provider in the chain until a successful response is received or all providers fail.

If all providers fail, a RuntimeError is raised. Each provider has its own loader function that returns None if the required API key is not set, allowing for easy configuration of providers through environment variables.