What's your rule for when an LLM call should NOT be retried?
Everyone has a retry story. Mine: an agent that retried a malformed request eleven times before anyone noticed, each attempt replaying the full conversation. The request was never going to succeed — a 400 doesn't heal — but the retry policy didn't care, because it only looked at "did it throw?" Our working rules now: Retry timeouts, 429s and 5xx — with backoff, jitter, and the server's…
Everyone has a rule for when to stop retrying an LLM call. Mine is simple: don't retry a 400, 401, or 403 error. The request is inherently flawed, so continuing to send it is pointless. Instead, surface the problem and fix the request. I also enforce a retry budget, allocating a portion of the task's total budget for retries. Once the budget is exhausted, the task fails honestly.
I never append to a half-generated answer after a mid-stream failure; instead, I restart the message cleanly. The one area I'm still unsure about is how to handle provider 529 errors or other overloaded conditions during peak times. Should I retry in place, or fail over to a different model and accept a potential drop in quality?
This particular error taught me the importance of having a nuanced retry strategy that considers the specific nature of the error and its potential impact on the user experience.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
