Fix Silent 30s AI Stream Timeouts with a Client Watchdog

AI streams can suddenly stop working after exactly 30 seconds due to a proxy timeout issue. The model itself does not stop generating responses; instead, the proxy treats a pause as a dead connection. This behavior is caused by the proxy_read_timeout setting in reverse proxies like Nginx. When the upstream server stops sending data for 30 seconds, the proxy closes the socket without sending a response, causing the client's fetch stream to end abruptly.

The original article describes a testing scenario where a chat widget rendered a spinner continuously, showing no error or close event, and then went silent after 30 seconds. No error messages or status changes were observed. To isolate the problem, the author tested the issue by bypassing the proxy and directly hitting the provider, confirming that the model was functioning correctly.

They also built a minimal local proxy that simulated a pause, resulting in a timeout when routed through the gateway. The root cause was identified as the gateway enforcing a 30-second proxy_read_timeout, which measures the gap between consecutive reads from the upstream. If the gap exceeds 30 seconds due to a thinking pause or tool waiting, the gateway closes the socket.

To address this issue, the author suggests adding a client-side watchdog to the streaming loop. The watchdog resets on every chunk received and triggers a stalled state if no chunk arrives within a specified threshold. If the silence lasts beyond the known timeout, it offers a recovery option. By implementing this watchdog, the UI can transition to a visible stalled state instead of a perpetually spinning spinner, providing users with a more honest representation of the stream's status.