everyais
← All docs

Guides

Streaming

Updated

Streaming sends the response as it is generated, so you can show the first tokens right away. Use it where a person is waiting, such as a chat screen.

Try it

Use the environment variables from the quickstart. Add stream: true, plus stream_options.include_usage: true for token usage at the end.

Python

python
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["EVERYAIS_API_KEY"],
    base_url="https://api.everyais.com/v1",
    timeout=240.0,
)
stream = client.chat.completions.create(
    model=os.environ["EVERYAIS_MODEL"],
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
    stream_options={"include_usage": True},
)
try:
    for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="", flush=True)
        if chunk.usage:
            print("\nUsage:", chunk.usage.total_tokens)
finally:
    stream.close()

Node.js

javascript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.everyais.com/v1",
  apiKey: process.env.EVERYAIS_API_KEY,
  timeout: 240_000,
});
const stream = await client.chat.completions.create({
  model: process.env.EVERYAIS_MODEL,
  messages: [{ role: "user", content: "Hello!" }],
  stream: true,
  stream_options: { include_usage: true },
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.log("\nUsage:", chunk.usage.total_tokens);
}

Handle these cases

  • The final usage chunk has an empty choices array. Check it before you read choices[0].
  • A Chat stream ends with data: [DONE]. If the connection closes without it, the response was cut off.
  • An error can arrive as an event after the stream started with HTTP 200. Treat error events and SDK exceptions as failures.
  • To stop generation, close the stream (stream.close() in Python, an AbortController in Node.js). The provider may not stop right away, and those tokens can still be billed.

Parsing SSE yourself

  • Events are data: {...} lines, and a blank line ends each event. Buffer network chunks until you reach one.
  • Lines starting with a colon, such as : ok, are keep-alive comments. Skip them.
  • Decode UTF-8 with a streaming decoder, since characters can be split across chunks.

Messages and Responses streams

These use different events from Chat. Use each SDK's stream helpers instead of Chat [DONE] handling.

APIMain events
/v1/messagesmessage_start, content_block_delta, message_delta, message_stop
/v1/responsesresponse.output_text.delta, response.completed

With background: true, Responses returns a job instead of a stream. See async jobs.

Retrying a stream

Idempotency-Key can't replay a finished stream; resending the same key returns 409. Before you retry, check the output you already received and your usage. With web search, citations and the search count arrive at the end. See web search.

Contact support