LLM streaming fallback: after the first token, the client owns the failure
A model gateway can switch providers cleanly until it has sent something to your client. After that point, every option changes an answer someone has already started reading. Gateways document that point in different places, and "first byte", "first frame" and "first token" are not the same moment in a stream. Our position: for agent workloads, the default after any content reaches the client…
When streaming, the client assumes responsibility for any failures that occur after the first token is sent. This point varies between response headers, first byte, first frame, and first token. The gateway should document this boundary and the application should handle retries accordingly. This advice applies to agent workloads, but streaming may be unsuitable for tool calls.
Gateway-side continuation works well for prose but not for tool calls. It is crucial to document and test the gateway against a contract, as documentation does not guarantee what the client receives.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
