A rate limit that counts the tokens you asked for, not the ones you got
Spent a while assuming a 413 meant my prompt was too long. It wasn't. The prompt was 20 tokens. Groq's free tier caps you at 8,000 tokens per minute, and that budget is charged against the max_tokens you declare , not the number the model actually generates. So this fails: prompt: 20 tokens, max_tokens: 8192 -> 413 "on tokens per minute (TPM): Limit 8000, Requested 8271" Nothing was generated.…
The rate limit imposed by Groq's API is based on the maximum number of tokens you request, not the number the model actually generates. This means that if you declare a higher limit than necessary, you risk hitting the rate limit even if the model produces fewer tokens. In the case described, a prompt with 20 tokens and a max_tokens request of 8192 resulted in a 413 error due to exceeding the 8,000 tokens per minute (TPM) limit.
This issue was consistent across several models tested, indicating it is not a problem with the model size, but rather a limitation of the API. It is crucial to set the max_tokens parameter to the exact amount needed, or risk the call failing before the model even processes the prompt. Additionally, the rate limit is a rolling window shared among all models, so a model that passed one request can still be limited on a subsequent one.
Providers like Alibaba clearly state whether they charge for input and output tokens separately, whereas Groq's documentation does not make this distinction.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
