Streaming

Send "stream": true and the answer arrives as server-sent events while the model writes it, instead of in one piece at the end. Each route streams in its own SDK's format, so the SDK's streaming loop reads it unchanged.

# -N prints each event as it arrives; the stream ends with data: [DONE]
curl -N https://api.vultrinference.com/v1/chat/completions \
  -H "Authorization: Bearer $VULTR_INFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4.1-flash",
    "stream": true,
    "messages": [{"role": "user", "content": "Count from one to five."}]
  }'

import os

from openai import OpenAI

client = OpenAI(base_url="https://api.vultrinference.com/v1", api_key=os.environ["VULTR_INFERENCE_API_KEY"])

stream = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    stream=True,
    messages=[{"role": "user", "content": "Count from one to five."}],
)
for chunk in stream:
    if chunk.usage:
        # The last chunk carries the usage, always
        print(f"\n{chunk.usage.prompt_tokens} in, {chunk.usage.completion_tokens} out")
    elif chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.vultrinference.com/v1", apiKey: process.env.VULTR_INFERENCE_API_KEY });

const stream = await client.chat.completions.create({
  model: "deepseek-v4.1-flash",
  stream: true,
  messages: [{ role: "user", content: "Count from one to five." }],
});
for await (const chunk of stream) {
  if (chunk.usage) {
    // The last chunk carries the usage, always
    console.log(`\n${chunk.usage.prompt_tokens} in, ${chunk.usage.completion_tokens} out`);
  } else {
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
}

Chat completions#

Each event is one chunk on a data: line, and the stream ends with data: [DONE]. This is the whole stream of a short answer, "hi there", with each chunk's id, model and other fixed fields left out, and the usage shortened:

data: {"choices":[{"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"choices":[{"delta":{"content":"hi"},"finish_reason":null}]}
data: {"choices":[{"delta":{"content":" there"},"finish_reason":"stop"}]}
data: {"choices":[{"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":8,...}}
data: [DONE]

• A delta holds only what it adds, and never a field set to null. Thinking arrives the same way, as reasoning.
• A tool call arrives in pieces of tool_calls: its id and name first, then its arguments, matched by index.
• The last chunk always carries usage, whether you asked for it or not. Its one choice has an empty delta, so a loop that reads chunk.choices[0] keeps working.

Responses#

Each event is an event: line and a data: line, numbered by sequence_number, in this order:

The pieces of the answer come in three kinds of delta:

There is no [DONE]. The last event says how the response ended: response.completed, response.incomplete when max_output_tokens cut it short, or response.failed.

Messages#

/v1/messages streams Anthropic's events, in this order, with one start, run of deltas and stop for each block of the answer:

message_delta carries the stop_reason.

When a stream fails#

A request that fails before the first token is answered with the real status and an ordinary error body, never a stream: check the status before you read events.

A failure after the stream has started cannot change the status, which was 200. Each route ends the stream with an error of its own kind instead:

Treat the text received before it as incomplete. Errors shows the chunk.

Other streams#

• Speech: with stream_format: "sse" the audio arrives in speech.audio.delta events, and speech.audio.done ends with the usage. Only pcm is streamed as it is made; other formats arrive in one piece.
• Transcriptions: with stream: true the text arrives in transcript.text.delta events, and transcript.text.done holds all of it.
• Music: a song can be heard while it renders, from its job's stream_url (Async jobs).