# Realtime transcription

Stream live audio over a WebSocket and read the transcript while the speaker is still talking, in OpenAI's transcription session protocol.

`wss://api.vultrinference.com/v1/realtime?intent=transcription` turns audio you stream into text you read as it is spoken, in OpenAI's transcription session protocol. It is for speech to text models that list `/v1/realtime` in their `endpoints` on [`GET /v1/models`](/docs/models), today `voxtral-mini-4b-realtime`, whose transcript trails the speaker by about 480 ms. A recorded file is still `POST /v1/audio/transcriptions`.

## Connect

```text
wss://api.vultrinference.com/v1/realtime?intent=transcription&model=voxtral-mini-4b-realtime
Authorization: Bearer $VULTR_INFERENCE_API_KEY
```

- `intent=transcription` is required; without it the answer is a 400 before the socket opens.
- Send the key as `Authorization: Bearer` or `x-api-key`, from a server. A key in the query string is refused with a 400, and a browser cannot set the header, so there is no browser flow in this version.
- A bad key is a 401 in the usual [error envelope](/docs/concepts/errors), before the upgrade.
- `model` may be in the URL or in `session.update`. With it in the URL, an unknown model (404) and a reached [limit](#limits) (429 with `Retry-After`) are HTTP answers before the socket opens. Otherwise they arrive as `error` events.
- The first event is `session.created`.

## Configure the session

```json
{"type": "session.update", "session": {"type": "transcription", "audio": {"input": {
  "format": {"type": "audio/pcm", "rate": 24000},
  "transcription": {"model": "voxtral-mini-4b-realtime"},
  "turn_detection": null
}}}}
```

| Field under `session.audio.input` | Accepted |
| --- | --- |
| `transcription.model` | Required before you send audio, unless `model` was in the URL |
| `format` | `audio/pcm` at 24000 Hz, OpenAI's only PCM rate. `rate: 16000` also works and skips a conversion |
| `turn_detection` | `null`: you decide where a turn ends by committing |
| `transcription.delay` | Only the model's own delay, 480 ms |

These are refused with an `error` event whose `param` names the field, and the session keeps the settings it had: `audio/pcmu` and `audio/pcma`, any other sample rate, a `turn_detection` that is not `null`, `transcription.language`, `languages`, `prompt` and `keywords`, `noise_reduction`, `include`, and a different `transcription.delay`. An accepted update is answered with `session.updated`.

## Send audio, read text

Audio is base64 PCM16, mono, little endian, at the session's rate.

```json
{"type": "input_audio_buffer.append", "audio": "<base64>"}
{"type": "input_audio_buffer.commit"}
```

- `append` has no answer. Text arrives as `conversation.item.input_audio_transcription.delta` events with a `delta`.
- `commit` ends a turn. It is answered with `input_audio_buffer.committed` (`item_id`, `previous_item_id`), and the turn ends with `conversation.item.input_audio_transcription.completed` carrying the `transcript` and `usage: {"type": "duration", "seconds": 4.12}`, the turn's audio to two decimals.
- `input_audio_buffer.clear` drops the turn's text and is answered with `input_audio_buffer.cleared`.
- One turn runs at a time. Audio you send during a turn is held until its `completed` event.
- A turn that runs long is ended for you, as server-side turn detection would: at the first pause after 4 minutes of audio, or at 5 minutes. You get `input_audio_buffer.committed` and `completed` for it without having committed, and the audio that follows starts the next turn. The model slows as a turn grows, so this keeps a long session in real time.
- Every event has an `event_id`. An error caused by an event of yours carries its id in `error.event_id`.

## Errors

An `error` event is `{"error": {"type", "code", "message", "param", "event_id"}}`. A turn the engine failed ends with `conversation.item.input_audio_transcription.failed` instead, and the socket stays open: the next `append` starts a new turn.

| `code` | Meaning |
| --- | --- |
| `session_not_configured` | Audio arrived before a model was set |
| `invalid_audio` | The base64 did not decode, or it was an odd number of bytes; that chunk is dropped |
| `model_not_found` | The model does not exist or does not serve realtime |
| `rate_limit_exceeded` | A [limit](#limits) is reached |
| `buffer_full` | More than 30 seconds of your audio is waiting behind a turn the model has not finished; audio is dropped until it catches up |
| `idle_timeout` | No audio for 5 minutes; the socket closes with 1000 |
| `session_expired` | A session lasts 60 minutes; sent at the end of the current turn, then the socket closes with 1000 |
| `server_restarting` | The service is restarting; the socket closes with 1012, so reconnect |

The service pings every 20 seconds; any WebSocket client answers without code of yours.

## Billing

You pay for the audio you send, per second, at the model's price on [its page](/docs/models), the same as a file: silence counts and so does the audio of a turn you cleared. A session is rounded up once, to a whole second, when it ends, not turn by turn. Usage appears in your account like any other transcription's.

## Limits

Each subscription may open 30 sessions a minute and hold 5 open at once per model, unless the model's page says otherwise. Past either, the answer is a 429 with `Retry-After` (or `rate_limit_exceeded` as an event), and a session holds its place until its socket closes.

## Not in this version

Browsers and ephemeral tokens, telephony formats (`audio/pcmu`, `audio/pcma`), voice activity detection by the server, language and prompt hints, and `intent=realtime` conversation sessions.