Streaming Responses on EXAIV: Handling Incremental Output in Production

Streaming changes the character of an API call. Instead of one response after a pause, you receive a sequence of partial results over time — which feels dramatically better in a chat interface and is considerably harder to build correctly. The hard parts are not the happy path: they are the moment a stream ends early, the moment a proxy buffers everything into one lump at the end, and the moment your code treats a truncated answer as a complete one.

This guide covers how streaming works against the EXAIV endpoint, how to consume it in Python and Node.js, how to assemble partial output correctly, and how to detect the three failure modes that make streaming look broken when the gateway is fine.

What a streaming response looks like

The endpoint returns text/event-stream. Each event is a block of lines, and the payload is a JSON object on a data: line:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"The "},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"short answer"},"finish_reason":"stop"}]}

data: [DONE]

Three fields matter for a client:

The delta object is the difference between streaming and non-streaming. In a normal response the text is in choices[0].message.content; in a chunk it is in choices[0].delta.content. Code written against message will appear to work in a streaming test — returning empty strings — because the field simply is not there.

Streaming from the command line

Start with curl. It shows the raw wire format, which is worth knowing when something misbehaves:

export EXAIV_API_KEY="YOUR_EXAIV_API_KEY"

curl -sS -N "https://exaiv.com/api/ai/v1/chat/completions" \
  -H "Authorization: Bearer ${EXAIV_API_KEY}" \
  -H "content-type: application/json" \
  -d '{
    "model": "gpt-6-sol",
    "messages": [{"role": "user", "content": "Write a two-sentence summary of rate limiting."}],
    "max_tokens": 256,
    "stream": true
  }'

The -N flag disables curl's output buffering, which matters — without it curl may hold output until the stream closes, making a working stream look like a non-streaming one.

Streaming in Python

The OpenAI SDK exposes the iterator directly. The one rule that matters is to consume the iterator inside the request context, so the connection is closed promptly:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_EXAIV_API_KEY",
    base_url="https://exaiv.com/api/ai/v1",
)

with client.chat.completions.create(
    model="gpt-6-sol",
    messages=[{"role": "user", "content": "Write a two-sentence summary of rate limiting."}],
    max_tokens=256,
    stream=True,
) as stream:
    parts = []
    for chunk in stream:
        delta = chunk.choices[0].delta
        if delta.content:
            parts.append(delta.content)
            print(delta.content, end="", flush=True)

    text = "".join(parts)

Accumulate the chunks yourself. The SDK does not join them for you, and code that renders only the last chunk will show a flicker rather than the full answer.

Streaming in Node.js

const client = new OpenAI({
  apiKey: process.env.EXAIV_API_KEY,
  baseURL: "https://exaiv.com/api/ai/v1",
});

const stream = await client.chat.completions.create({
  model: "gpt-6-sol",
  messages: [{ role: "user", content: "Write a two-sentence summary of rate limiting." }],
  max_tokens: 256,
  stream: true,
});

let text = "";
for await (const chunk of stream) {
  const delta = chunk.choices[0].delta;
  if (delta.content) {
    text += delta.content;
    process.stdout.write(delta.content);
  }
}

Both examples follow the same shape: read chunks, keep the text, and decide separately what to do if the stream ends early.

Choosing streaming or not

Streaming is not always the better choice, and picking it by default adds complexity without benefit.

Two specific arguments for non-streaming: the response arrives as one payload, which removes every partial-state question from your code; and a structured response can be parsed the moment it arrives rather than being accumulated across chunks.

The argument for streaming is usually just latency perception, and it is a strong one — but it is a reason to stream the interactive path specifically, not every path.

Assembling chunks correctly

Concatenation is the easy part. Three details cause real bugs:

• Only append non-empty deltas. Many events carry delta.content as an empty string — role-only events, usage events, keepalive events. Appending them is harmless, but branching on their presence is not.
• Do not assume one chunk per token. A chunk can contain several tokens, and a token can be split across chunks when a multi-byte character is split. Appending the raw strings in order reproduces the original text exactly; do not trim or normalize between chunks.
• A finish_reason of length means truncation. The model stopped because it hit max_tokens, not because the answer was complete. Treat the output as a partial answer and either raise the limit or surface the truncation to the user.

Detecting a truncated stream

This is the failure worth designing for. A stream can end without the [DONE] sentinel when the connection drops, an intermediary times out, or the process is killed mid-flight. The client sees a clean end-of-iteration and may treat a partial answer as a finished one.

The robust pattern is to track whether the stream completed:

completed = False
parts = []

with client.chat.completions.create(..., stream=True) as stream:
    for chunk in stream:
        delta = chunk.choices[0].delta
        if delta.content:
            parts.append(delta.content)
        if chunk.choices[0].finish_reason:
            completed = True

if not completed:
    # The stream ended early: the answer may be incomplete.
    # Decide explicitly: retry, prompt the user to continue, or mark the result partial.
    pass

The same check applies at the HTTP level: if you read the raw event stream yourself, look for the literal data: [DONE] sentinel. Its absence is the clearest available signal that the response did not finish normally.

Buffering defeats streaming

The most common report is "streaming does not stream". The gateway is fine and the model is fine; something between the client and the gateway accumulated the whole response and delivered it in one piece.

Symptom and causes:

The first check is the path between client and gateway. A proxy that does not stream will force a non-streaming response even though the request asked for stream: true. In a self-hosted environment, confirm that the reverse proxy configuration passes Cache-Control: no-cache and that response buffering is disabled for this route. If you cannot control the path, a non-streaming request with a generous timeout is a more honest choice than streaming that arrives all at once.

When a stream fails mid-answer

A stream can break in several distinguishable ways, and each one implies a different response.

Two design decisions make these cases manageable. First, record how the stream ended — completion flag, finish reason, or error — so the failure is visible in logs rather than reconstructed later. Second, decide in advance what the user sees for a partial answer, because that decision is always made under pressure if it was not made in advance.

A partial answer labelled as partial is a much smaller problem than a partial answer presented as complete. It costs the user one sentence of explanation and saves you a bug report about a broken feature.

Timeouts, cancellation, and concurrency

A streaming request occupies a connection for as long as the model takes to produce the answer. Two settings need explicit thought:

• Read timeout. Set it to a value longer than the slowest acceptable generation. A stream that never terminates otherwise holds the connection until the operating system gives up.
• Client cancellation. When the user navigates away or the request is superseded, close the stream. Not closing it leaves the upstream request running and consuming resources for an answer nobody will read.

Concurrency matters more with streaming than with non-streaming, because each request holds a connection rather than returning promptly. If the upstream account has a concurrency limit, a burst of streaming requests will hit it sooner than the same number of non-streaming requests. Size the pool for the streaming case, and read the rate limits and concurrency guide before raising the limits.

Streaming checklist

• The client reads the response body incrementally rather than buffering the whole payload.
• Text is read from choices[0].delta.content, with message.content not used on the streaming path.
• Chunks are accumulated into a single string in arrival order.
• The stream is checked for completion — the finish_reason or the [DONE] sentinel — and partial output is handled explicitly.
• A finish_reason of length is treated as truncation, not completion.
• A read timeout is configured, and the stream is closed on cancellation.
• The network path has been checked for a buffering proxy.
• Concurrency limits are sized for long-lived connections.

Further reading

• GPT Model Selection Guide — choosing a tier before you tune how its output is consumed.
• Chat Completions Parameters Reference — the request fields that control generation, including stream.
• Error Handling and Retry Guide — what to do when a stream fails instead of finishing.

FAQ

Related reading

• Qwen Streaming and Tool Calling
• JSON Structured Output Guide
• Qwen API Troubleshooting