What is an LLM gateway? A practical guide for side projects

An LLM gateway is a single API endpoint that acts as a middleman for multiple model providers. When you send a request to the gateway, it handles the provider keys, routing, retries, and usage records. Instead of needing a separate account, key, and request format for each provider, the gateway simplifies this to one base URL and one key. This centralization allows you to see which model answered a request from a single location.

However, a gateway does not improve the capabilities of a weak model. For tasks requiring long context, tool calling, or code editing, the selected model must still support these requirements. The gateway's routing function simply directs requests to the most appropriate provider, but it does not enhance the model's inherent abilities.

Before committing to a gateway, there are several factors to consider. First, ensure the model list is public and clearly distinguishes between free and paid models, including what happens if a free model runs out of capacity. If a provider fails, the gateway should automatically route the request to the next candidate or return an error to your application, rather than leaving your app in the lurch.

Additionally, the gateway's API should be compatible with existing clients like OpenAI or Anthropic's, allowing you to switch providers without extensive code changes. Lastly, the ability to view usage metrics, such as per-key and per-day data, is crucial for monitoring and budgeting purposes.

A practical way to start using a gateway is to begin with a free tier and a small task. This allows you to gauge how the gateway performs when the free allowance is exhausted. If the gateway seamlessly switches to paid capacity, it's essential to set a budget in advance. The true test of a gateway's utility lies in how it handles failures and unexpected situations, not just what it claims on its landing page.