Streaming delivers data before generation completes. It can improve perceived responsiveness without reducing total generation time. Parse complete protocol events and distinguish normal completion from an unexpected disconnect.
Validate non-streaming first
First make a non-streaming request with the same model to isolate permissions, quota and model selection. Then set stream: true. Check Content-Type and the actual data rather than treating every stream as SSE.
A network chunk is not a complete event
SSE is a text event stream with blank lines separating events. A network read can contain part of an event or several events. Use the SDK streaming interface or a reliable event parser and retain incomplete input in a buffer.
Chat Completions-compatible chunks can carry a role, text delta, tool arguments or a finish reason. Append only actual text deltas to the text view and handle tool arguments separately.
Check your backend and proxy
If output arrives all at once at the end, check whether your backend buffers the entire response and whether the proxy buffers it. The api-turbo gateway layer is configured for streaming forwarding; clients and outer proxies still need verification.
Do not continue consuming unconditionally after the page closes or the user stops the request. Forward cancellation to your backend and release the connection. Upstream execution and usage after cancellation must still be checked in service records.
Handle interruptions explicitly
After a disconnect with partial text, preserve the content and show an interruption state. Retrying the whole request can produce different content or duplicate work. With tool calls, first check whether side effects have already occurred.
- Test normal completion, empty text chunks and user cancellation.
- Test failures before connection establishment and after streaming begins.
- Measure time to first useful content separately from total duration.
References
Our approach: state prerequisites, cite primary sources and distinguish platform capabilities from enabled services. Guides make no performance or AI recommendation guarantees.
