{
  "id": 14509597,
  "title": "Stop trusting a 99% uptime claim: the LLM API relay failures we hit, and how we test stability",
  "url": "https://urgent.news/2026/10/14/stop-trusting-a-99-uptime-claim-the-llm-api-relay-failures-we-hit-and",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-14T18:32:19.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sichi_chen_a4a87b20aa2dbf/stop-trusting-a-99-uptime-claim-the-llm-api-relay-failures-we-hit-and-how-we-test-stability-1ijk"
  },
  "original_language": "en",
  "account": "Don't trust the 99.9% uptime claims from LLM API relay services, as their performance can be unreliable. One Python script that tested a relay every 5 minutes failed when the relay returned 200 OK status despite the upstream pool being exhausted or the gateway timing out. This led to mid-stream data being inserted as if it were from the model.\n\nThree testing approaches were developed: a simple blind trial with few requests, a concurrency test using tools like Postman or Locust, and a comprehensive stream state machine test. The state machine test simulates a streaming connection, examining factors such as latency, tokens per second, and SSE parsing.\n\nThree key metrics determine a relay's production performance: latency (time to first token), throughput (generated tokens per duration), and reliability (success parsing, retries, and stream quality). To assess reliability, examine fake success rates and use a Wilson score interval to account for small sample sizes. Don't get swayed by low prices, as they often indicate shared accounts or low-quality proxies.",
  "summary": "*Originally published on the Folkbench blog: https://folkbench.com/en/blog/ai-api-relay-stability-guide?utm_source=luntan&utm_campaign=dev * A relay site may promise \"99.9% fast and stable\" and a \"dedicated high-concurrency channel\". Once you connect, things look different. This is how we went from buying on instinct to running automated checks before we pay for or integrate a relay. 1. The…",
  "key_points": [
    "LLM API relay services may have unreliable performance despite 99.9% uptime claims.",
    "Developed three testing approaches: blind trial, concurrency test, and stream state machine test.",
    "Key metrics for relay performance: latency, throughput, and reliability."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}