{
  "id": 18320177,
  "title": "LLM streaming fallback: after the first token, the client owns the failure",
  "url": "https://urgent.news/2026/11/05/llm-streaming-fallback-after-the-first-token-the-client-owns-the",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-05T14:41:35.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/worksi_25025494650/llm-streaming-fallback-after-the-first-token-the-client-owns-the-failure-ici"
  },
  "original_language": "en",
  "account": "When streaming, the client assumes responsibility for any failures that occur after the first token is sent. This point varies between response headers, first byte, first frame, and first token. The gateway should document this boundary and the application should handle retries accordingly. This advice applies to agent workloads, but streaming may be unsuitable for tool calls. Gateway-side continuation works well for prose but not for tool calls. It is crucial to document and test the gateway against a contract, as documentation does not guarantee what the client receives.",
  "summary": "A model gateway can switch providers cleanly until it has sent something to your client. After that point, every option changes an answer someone has already started reading. Gateways document that point in different places, and \"first byte\", \"first frame\" and \"first token\" are not the same moment in a stream. Our position: for agent workloads, the default after any content reaches the client…",
  "key_points": [
    "After the first token, client owns streaming failure",
    "Document gateway boundary, handle retries accordingly",
    "Streaming unsuitable for tool calls, test against contract"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}