Realtime transcription

wss://api.vultrinference.com/v1/realtime?intent=transcription turns audio you stream into text you read as it is spoken, in OpenAI's transcription session protocol. It is for speech to text models that list /v1/realtime in their endpoints on GET /v1/models, today voxtral-mini-4b-realtime, whose transcript trails the speaker by about 480 ms. A recorded file is still POST /v1/audio/transcriptions.

Connect#

wss://api.vultrinference.com/v1/realtime?intent=transcription&model=voxtral-mini-4b-realtime
Authorization: Bearer $VULTR_INFERENCE_API_KEY

• intent=transcription is required; without it the answer is a 400 before the socket opens.
• Send the key as Authorization: Bearer or x-api-key, from a server. A key in the query string is refused with a 400, and a browser cannot set the header, so there is no browser flow in this version.
• A bad key is a 401 in the usual error envelope, before the upgrade.
• model may be in the URL or in session.update. With it in the URL, an unknown model (404) and a reached limit (429 with Retry-After) are HTTP answers before the socket opens. Otherwise they arrive as error events.
• The first event is session.created.

Configure the session#

{
  "type": "session.update",
  "session": {
    "type": "transcription",
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "transcription": {
          "model": "voxtral-mini-4b-realtime"
        },
        "turn_detection": null
      }
    }
  }
}

These are refused with an error event whose param names the field, and the session keeps the settings it had: audio/pcmu and audio/pcma, any other sample rate, a turn_detection that is not null, transcription.language, languages, prompt and keywords, noise_reduction, include, and a different transcription.delay. An accepted update is answered with session.updated.

Send audio, read text#

Audio is base64 PCM16, mono, little endian, at the session's rate.

{"type": "input_audio_buffer.append", "audio": "<base64>"}
{"type": "input_audio_buffer.commit"}

• append has no answer. Text arrives as conversation.item.input_audio_transcription.delta events with a delta.
• commit ends a turn. It is answered with input_audio_buffer.committed (item_id, previous_item_id), and the turn ends with conversation.item.input_audio_transcription.completed carrying the transcript and usage: {"type": "duration", "seconds": 4.12}, the turn's audio to two decimals.
• input_audio_buffer.clear drops the turn's text and is answered with input_audio_buffer.cleared.
• One turn runs at a time. Audio you send during a turn is held until its completed event.
• A turn that runs long is ended for you, as server-side turn detection would: at the first pause after 4 minutes of audio, or at 5 minutes. You get input_audio_buffer.committed and completed for it without having committed, and the audio that follows starts the next turn. The model slows as a turn grows, so this keeps a long session in real time.
• Every event has an event_id. An error caused by an event of yours carries its id in error.event_id.

Errors#

An error event is {"error": {"type", "code", "message", "param", "event_id"}}. A turn the engine failed ends with conversation.item.input_audio_transcription.failed instead, and the socket stays open: the next append starts a new turn.

The service pings every 20 seconds; any WebSocket client answers without code of yours.

Billing#

You pay for the audio you send, per second, at the model's price on its page, the same as a file: silence counts and so does the audio of a turn you cleared. A session is rounded up once, to a whole second, when it ends, not turn by turn. Usage appears in your account like any other transcription's.

Limits#

Each subscription may open 30 sessions a minute and hold 5 open at once per model, unless the model's page says otherwise. Past either, the answer is a 429 with Retry-After (or rate_limit_exceeded as an event), and a session holds its place until its socket closes.

Not in this version#

Browsers and ephemeral tokens, telephony formats (audio/pcmu, audio/pcma), voice activity detection by the server, language and prompt hints, and intent=realtime conversation sessions.