Designing LLM Fallbacks So Your App Survives a Provider Outage
What Happens When Your LLM Provider Goes Down? Fallback Design for Indie SaaS Your AI feature worked perfectly in the demo, then one Tuesday your provider returns 503s for forty minutes and every model call becomes a spinner. The uncomfortable truth for a one-person company is that you cannot control provider uptime — you can only control what your product does during the gap. This guide is about…
When your AI feature works well in a demonstration, it can be disheartening to discover that a provider outage renders it unusable for several minutes. As a small team, you cannot control the availability of the provider, but you can control how your product behaves during the gap. This article outlines the four failure modes that affect small teams, the retry and fallback layers that address them, and how to build all of it without significantly increasing costs.
The primary recommendation is to assume that every provider will experience intermittent failures and design your system accordingly. Implement four layers of defense: strict timeouts, retries with jittered backoff for idempotent calls, a two-provider fallback chain behind a gateway abstraction, and a graceful degradation path that maintains the rest of your product's functionality.
During a provider outage, partial degradation is often the most common issue. One model or one region may fail while the rest of the API remains healthy. This can result in an error rate that is worse than a full outage because it goes unnoticed in logs. Additionally, latency spikes may occur before errors, and rate-limit storms can arise when retries pile up, causing your own quota to be exceeded.
The cheapest and most effective fix for request-hungry scenarios is to implement explicit timeouts ranging from 15 to 30 seconds for interactive features and longer for background jobs. Most SDK defaults are too generous, and the absence of a timeout can turn a provider hiccup into a product outage.
When retries are necessary, they should be disciplined. Only retry on connection errors, timeouts, 429 responses, and 5xx errors. Avoid retrying 400 responses or content-policy rejections, as they will likely fail again and incur additional costs. Use exponential backoff with jitter, cap attempts at two or three, and set a total retry budget per user action to prevent a burst of ten calls from occurring due to an impatient click.
A fallback chain is essential when one provider fails after its retries. This involves sending the request to a second provider through a single interface rather than directly integrating it into your feature code. An LLM gateway, whether self-hosted or hosted, is crucial for establishing this interface. The key design decisions include pairing providers with different infrastructure and avoiding silent model swaps.
If a fallback chain is implemented, a circuit breaker should be used as the third piece of the puzzle. After N consecutive failures, the circuit breaker should stop sending traffic to the unhealthy provider for a cooldown period and immediately route requests to the fallback provider. This prevents all requests from paying the primary provider's timeout before failing over.
In situations where a provider outage is unavoidable, graceful degradation is the best approach instead of failing loudly. Determine in advance which features are worth waiting for and which are not. Blocking features should show an honest error with a retry button, never leaving an infinite spinner. Enhancement features should fall back to non-AI paths, such as displaying the document when summarization is unavailable.
Background features can be enqueued and processed when capacity returns, keeping your users informed and minimizing disruption.
By implementing these four layers of defense, you can mitigate the impact of provider outages and ensure that your app remains functional and user-friendly during outages. Although achieving 100% model availability may not be possible, focusing on user experience during outages will help maintain customer trust and satisfaction.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.