One Project Ate the Shared Free Tier: A Per-Project Quota Pattern for LLM Gateways
Three projects shared one gateway, one API key, and one 10-million-token allowance. On day nineteen, a batch job that summarized support tickets consumed 7.1 million tokens in four hours, and every interactive request from the other two projects started failing with quota errors. The dashboard showed a single number, zero tokens remaining, and it did not say which project had spent them. The…
Three projects utilized a single gateway, API key, and a 10-million-token allowance. A batch job that summarized support tickets consumed 7.1 million tokens in four hours, causing all interactive requests from the other two projects to fail with quota errors. The dashboard displayed zero tokens remaining without indicating which project had consumed them.
The issue was not solely due to the batch job; the gateway lacked project-specific tracking, so every request drew from the same pool, and the first consumer to exhaust it won.
MonkeyCode introduced a per-project quota layer to address this issue while running services on their free model access, which includes a 10-million-token allowance. This layer was built using plain Python and works with any OpenAI-compatible endpoint. The key problem was the lack of an abstraction for projects. A project identifier was introduced, either as a header or a path prefix, allowing the quota layer to look up each project's budget based on the ID.
The daily and per-minute request limits for each project were set to 3,000,000 and 60, 4,000,000 and 120, and 2,000,000 and 30 tokens, respectively. These limits totaled 9 million tokens, leaving a 10% buffer for unexpected traffic. A pre-check was implemented to reject requests before spending any tokens, ensuring that a retry loop couldn't consume the entire allowance before any monitoring alert could fire.
This was achieved by estimating the token cost based on prompt and expected completion length and comparing it to the project's remaining budget.
After each request, the actual usage was reconciled with the estimate, ensuring the quota remained accurate even if the estimate was off. A three-step degradation ladder was introduced, warning clients at 80% of their daily budget, queuing requests up to 30 seconds at 100%, and rejecting them immediately with a Retry-After header over the per-minute cap. This ladder provided clients with opportunities to adapt before being rejected.
To survive restarts, the in-memory counters were stored in a SQLite table, with the current usage reset lazily on the first request of a new day.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
