What To Do When a Free LLM API Says 429 (A Field Playbook)
What To Do When a Free LLM API Says 429 (A Field Playbook) Every free LLM API says yes right up until the moment it says no. The no arrives as an HTTP 429, usually forty requests into a demo or halfway through a batch job, and most tutorials end the story at "add a retry loop." Here is what actually works when you are building on free tiers, provider by provider and layer by layer. First, find…
When a free language model API returns an HTTP 429 error, it signals that a rate limit has been exceeded. The specific cause depends on the type of limit that triggered the error. The 429 message can occur due to several factors:
1. Requests per minute (RPM): If the API is called too frequently, it will return a 429 error. Simply adding a retry loop is not enough to solve this issue.
2. Tokens per minute (TPM): If a single prompt or request consumes a large number of tokens, it can cause the API to hit the TPM limit. Shortening the context or limiting the maximum number of tokens can help avoid this problem.
3. Requests per day (RPD): If the number of API calls exceeds the daily limit, the API will return a 429 error. The only solution for this limit is to spread out the requests over a longer period or switch to a different provider.
4. Concurrency: Some APIs limit the number of concurrent requests they can handle. If more than the allowed number of requests are made simultaneously, a 429 error will be returned. In this case, queuing the requests instead of parallelizing them is the recommended approach.
Before implementing any code fixes, it is essential to read the response body and headers to determine which specific limit has been reached. Groq sends headers like x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens, while the Gemini API returns a RESOURCE_EXHAUSTED error with details about the violated window. OpenRouter provides similar information in its error text and headers. If no information is provided, it is best to treat all windows as closed and apply a backoff strategy.
The recommended retry strategy involves:
1. Honoring the retry-after header when present, as it accurately indicates when to retry the request.
2. Using exponential backoff with jitter for cases where retry-after is not present. This involves starting with a short delay (e.g., 1 second), doubling the delay for each subsequent attempt, and adding a random offset to avoid thundering herd problems. The delay should be capped at a few minutes.
3. When retrying a daily limit, it is important to stop after 3-5 attempts and surface the error, as continuously retrying will not help and may lead to unnecessary resource usage.
It is crucial to understand that a free provider's limits apply to a single quota pool. To handle this, one should treat free tiers as a pool of pools and fail requests to the next provider when a specific window is exhausted. By keeping multiple OpenAI-compatible API keys, you can route requests based on the type of limit that has been reached. This approach allows your client code to remain unchanged, as all providers use the same OpenAI chat format.
In summary, when building applications on free LLM APIs, it is essential to monitor and manage rate limits at the individual provider level. Understand the specific limits that apply to each provider, implement appropriate retry strategies, and consider using a provider abstraction layer to simplify the process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
