Stop trusting a 99% uptime claim: the LLM API relay failures we hit, and how we test stability

Don't trust the 99.9% uptime claims from LLM API relay services, as their performance can be unreliable. One Python script that tested a relay every 5 minutes failed when the relay returned 200 OK status despite the upstream pool being exhausted or the gateway timing out. This led to mid-stream data being inserted as if it were from the model.

Three testing approaches were developed: a simple blind trial with few requests, a concurrency test using tools like Postman or Locust, and a comprehensive stream state machine test. The state machine test simulates a streaming connection, examining factors such as latency, tokens per second, and SSE parsing.

Three key metrics determine a relay's production performance: latency (time to first token), throughput (generated tokens per duration), and reliability (success parsing, retries, and stream quality). To assess reliability, examine fake success rates and use a Wilson score interval to account for small sample sizes. Don't get swayed by low prices, as they often indicate shared accounts or low-quality proxies.