Redis Rate Limits for LLM API Keys and Tenant Quotas
This post was created with AI assistance and reviewed for accuracy before publishing. Rate limiting a normal API protects your servers. Rate limiting an LLM API protects your bank account. The difference matters, because it changes what you are counting and where the limit has to live. A conventional web endpoint costs you CPU time measured in milliseconds. A single model call can cost real…
Rate limiting an LLM API protects your bank account more than just normal API protection. Conventional web endpoints cost CPU time, while a single model call can cost real money. The key to managing this is counting tokens, not just requests. Rate limiting helps stop runaway loops quickly, but the token limit is what actually caps spend.
Design the whole security model around the tenant identifier, obtained from an authenticated session. The identifier should not come from the request body, as anyone could manipulate it. Include the operation in the key too to ensure different types of API calls are properly accounted for.
Implement a sliding window that does not drift, using a sorted set in Redis. Each request is a member scored by timestamp, and expired entries are trimmed on read. This avoids issues where a caller can send a full quota at one moment and another full quota at another moment.
Run the check as a Lua script in Redis to ensure atomicity. Trim, count, and insert all happen as one unit, preventing race conditions where two concurrent requests can both proceed when they should be limited. Set the sorted set to expire idle tenants on their own, so the key count tracks active tenants rather than every tenant ever known.
Reserve an estimate before the call and reconcile after, reserving the maximum output tokens as a pessimistic estimate. Most responses come in well under the ceiling, and any difference is refunded. This avoids overcharging tenants if they exceed their quota due to in-flight requests when they hit the limit.
Handle Redis outages by choosing appropriately between latency and spend. Limit Redis down on spend quotas, as it is an expensive outage to fail open. Monitor the limiter's own latency, keeping it cheap by placing Redis in the same region as the service and using pipelining for multiple checks in a single round trip.
Begin by implementing a per-tenant token quota on an hourly window, failing closed. This protects against the primary failure mode that hurts operations: unbounded usage.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
