A streaming model call can keep delivering useful output and still fail because one layer treats it as an ordinary short request. Before extending every timeout, identify which clock is expiring. This sprint walks through a practical setup for long responses, from the SDK to the proxy, with clear rules for stalled connections, cancellation, and retries.
1. Map every timeout on the streaming path
List the components between your application and the model: SDK, HTTP client, application server, reverse proxy, load balancer, and relay. Each can close the connection independently. Increasing an SDK timeout will not fix a shorter deadline enforced by your hosting platform.
- Connection timeout: Limits the wait to establish a connection.
- Read or idle timeout: Limits the gap between incoming data, depending on the client.
- Total deadline: Caps the full request lifetime, even while data continues arriving.
Record the effective settings, not just the values in your source code. Environment variables, SDK defaults, and infrastructure policies can change what actually runs.
2. Configure the streaming client explicitly
Use streaming mode and consume the response incrementally. With an OpenAI-compatible SDK, SwiftRelay uses https://api.fastapirelay.com/v1 as the base URL. Claude Code and Gemini CLI use the bare host, https://api.fastapirelay.com. These endpoint choices do not override the client's timeout behavior.
For a long generation with Claude Sonnet 5.5, for example, inspect the timeout configuration supported by your installed SDK version. Some clients accept separate connection and read limits; others expose a single request deadline. Confirm whether that deadline covers stream consumption or only the initial response.
Keep a bounded connection timeout, allow an appropriate read window, and retain an explicit overall deadline. Avoid disabling every limit: an abandoned connection should not hold resources indefinitely.
3. Separate stream activity from visible text
A response can arrive in phases: response headers, stream events, visible text, tool arguments, and completion metadata. Waiting for the first printable word is not the same as waiting for the first network data.
Track transport activity separately from user-visible progress. A valid event may indicate that the stream is alive without adding text to the interface. Conversely, an open socket alone does not prove that useful data is still arriving.
If you implement an idle watchdog, reset it on the activity your protocol defines as meaningful. Verify whether heartbeat events affect the underlying HTTP client's read timeout; an application-level timer and a transport-level timer need not behave identically.
4. Remove buffering and align proxy deadlines
If your backend forwards the stream to a browser, check that every intermediary supports incremental delivery. Proxy buffering can collect chunks instead of forwarding them, making an active upstream response appear stalled downstream.
Review response buffering, idle connection limits, and maximum request duration in your deployment. Forward events as they arrive, and use the appropriate content type for your streaming protocol. Do not assume that flushing application output bypasses a platform's hard duration limit.
Where that hard limit cannot accommodate the work, consider a background job with a separate progress channel rather than repeatedly raising client timeouts.
5. Cancel upstream work and retry deliberately
Connect the user's stop action, browser disconnect, and application deadline to cancellation of the upstream request. Otherwise, your server may continue reading and processing output nobody will receive. Cancellation propagation is best effort; do not assume it instantly stops all upstream work or billing.
Once partial output has reached the user, a retry is a new generation, not a transparent continuation. It may duplicate text, repeat tool actions, and consume additional tokens. Inspect automatic retry settings and avoid replaying requests with side effects unless those actions are protected against duplication.
6. Test timeout failures before the first production run
Test delayed initial data, a midstream pause, a downstream disconnect, and an overall deadline reached while events are still arriving. Log the terminating layer, error type, whether output was already delivered, and whether upstream cancellation was attempted. Keep credentials and sensitive prompt content out of logs.
SwiftRelay is a practical fit when you want a shared relay endpoint for supported models and OpenAI-compatible tooling, with per-token billing. Sign up for SwiftRelay, use the small gift credit for a controlled streaming test, and confirm your timeout behavior before sending production traffic.