Time lyrics and speech to their audio

POST /v1/audio/alignments takes a recording and the words it holds, and answers with when each word and each line is sung or spoken. Nothing is transcribed: every word that comes back is one you sent, so the text stays exactly yours. Use it to light up lyrics as a song plays, to cue captions, or to cut a recording at a line.

The model is aligner-1. It is synchronous: a three-minute song takes about 5 seconds, plus the upload. It costs $0.002 a minute of audio, billed by the second; its model page has the price and limits as they stand.

Align a recording#

This makes a short clip with the speech endpoint, then aligns it. The Python version also writes the lines out as SRT captions.

# A short clip to align: the speech endpoint reads the text aloud
curl -s https://api.vultrinference.com/v1/audio/speech \
  -H "Authorization: Bearer $VULTR_INFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-tts-12hz-1.7b", "voice": "vivian", "response_format": "mp3", "input": "The quick brown fox jumps over the lazy dog. It barked twice, then slept."}' \
  -o clip.mp3

# Align it with the words it holds, one line per line; speech mode skips separating a vocal from music
curl -s https://api.vultrinference.com/v1/audio/alignments \
  -H "Authorization: Bearer $VULTR_INFERENCE_API_KEY" \
  -F model=aligner-1 \
  -F file=@clip.mp3 \
  -F mode=speech \
  -F 'text=The quick brown fox jumps over the lazy dog.
It barked twice, then slept.' | jq '{lines: [.lines[] | {text, start, end, certain}], usage}'

import os

import requests

API = "https://api.vultrinference.com/v1"
headers = {"Authorization": f"Bearer {os.environ['VULTR_INFERENCE_API_KEY']}"}
text = "The quick brown fox jumps over the lazy dog.\nIt barked twice, then slept."

# A short clip to align: the speech endpoint reads the text aloud
clip = requests.post(f"{API}/audio/speech", headers=headers, json={
    "model": "qwen3-tts-12hz-1.7b",
    "voice": "vivian",
    "response_format": "mp3",
    "input": text.replace("\n", " "),
})
clip.raise_for_status()

# Align it; speech mode skips separating a vocal from music
response = requests.post(
    f"{API}/audio/alignments",
    headers=headers,
    data={"model": "aligner-1", "text": text, "mode": "speech"},
    files={"file": ("clip.mp3", clip.content, "audio/mpeg")},
)
response.raise_for_status()
alignment = response.json()

for word in alignment["words"]:
    print(f"{word['start']:6.2f} {word['end']:6.2f}  {word['text']}")


# Each line as an SRT caption
def stamp(seconds):
    ms = round(seconds * 1000)
    return f"{ms // 3600000:02}:{ms // 60000 % 60:02}:{ms // 1000 % 60:02},{ms % 1000:03}"


with open("captions.srt", "w") as f:
    for n, line in enumerate(alignment["lines"], 1):
        f.write(f"{n}\n{stamp(line['start'])} --> {stamp(line['end'])}\n{line['text']}\n\n")

print("duration", alignment["usage"])

The text you send#

Send the words in the order they are heard, one line of the song or script per line. The line breaks are yours to choose: each becomes a line in the answer, with its own start and end.

• Write out every repeat. A chorus sung three times is written three times. The words are placed in order, so a chorus written once is placed once.
• Label sections in square brackets. A line that is only [Chorus] or [Verse 2 - Singer B] names the lines under it and is never aligned. Each line in the answer carries its section's number and label.
• Write numbers as words. The model reads letters, so 21 has nothing to place; write twenty-one.
• Send only what is heard. Every word you send is placed somewhere, so a line that is not in the recording still gets times, usually with a low score. Leave it out instead. A pause or an instrumental break with no words is left empty, not stretched over.

Songs and speech#

mode is lyrics by default, which separates the vocal from the music before aligning; that is what a song needs. Send mode=speech for a podcast, an interview, narration or anything else without a backing track: it skips the separation, and separated in the answer is false.

The answer#

{
  "duration": 220.89,
  "separated": true,
  "lines": [
    {
      "index": 0,
      "section": 1,
      "tag": "Verse",
      "text": "The lamp was low, the door was barred with iron",
      "start": 21.22,
      "end": 25.66,
      "score": 0.981,
      "certain": true
    }
  ],
  "words": [
    {
      "line": 0,
      "text": "lamp",
      "start": 21.48,
      "end": 21.79,
      "score": 0.94
    }
  ],
  "unaligned": [],
  "usage": {
    "type": "duration",
    "seconds": 220.89,
    "cost": 0.00736593
  }
}

• lines has one entry per line of your text, in order, with times in seconds from the start of the audio. section counts the [Label] sections from 0 and tag is the label, or null before the first one.
• words has every word of every line, written as you wrote it, with the line it belongs to.
• score is how well the audio matched, from 0 to 1. certain is the model's own judgement that a line is placed right. It is usually right, but not always: on the benchmark below about 2% of lines were more than a second off and still marked certain. Check the lines that are not certain first.
• unaligned lists lines the model could not place at all, with the reason, instead of leaving them out silently.

How accurate it is#

On JamendoLyrics, a public set of 79 songs with every word's start marked by hand, half the words are placed within 0.04 seconds of where they are sung, and 92 to 98% within 0.3 seconds, in English, French, German and Spanish. On 69 minutes of meeting recordings in speech mode, 98.8% of words land within 0.3 seconds of the recordings' published timings, which were made by machine rather than by hand.

Languages#

aligner-1 reads the Latin alphabet: English, Spanish, French, German, Portuguese, Italian, Dutch, Turkish and the other languages its model page lists. Accented letters are read as their plain ones, so niño and Mädchen align, and come back as you wrote them.

Text in another script (Chinese, Japanese, Korean, Cyrillic, Arabic and others) has nothing the model can read, and the request is a 400 that says so. A language the model does not list is a 400 naming the ones it does. Neither is billed.

Billing and errors#

A call that returns lines bills the whole recording, rounded up to the second: 220.89 seconds bills 221. usage.cost is that charge in dollars. A refused request bills nothing.