Write prompts for MiniMax-H3

MiniMax-H3 makes a video with its sound already in it: speech, ambience, sound effects and music, all from one prompt of up to 8,000 characters. You can also give it a first frame, a last frame, or up to twelve reference images, videos and audio clips. Its model page has every field and the price.

MiniMax writes H3's prompts in a fixed layout of named fields, and publishes how in the h3-prompt-writing guide in the MiniMax-H3 repository. This page is that guide applied to this API: what to send, how the API hands your inputs to the model, and what our test renders through this API showed.

Make a video from a structured prompt#

This six-second clip is two shots with a line of dialogue, a cut at 3.5 seconds, ambient sound and a score.

# MiniMax-H3's layout: the shots with their dialogue, then the ambient sound, then the score
read -r -d '' prompt <<'PROMPT' || true
integrated_multimodal_description: [Shot 1] Live-action, cinematic, a medium shot frames an elderly fisherman in a knitted flat cap and an olive jacket sitting on a stone harbour wall at dawn, a green net across his knees and small wooden boats rocking in the water behind him. The camera pushes in with small amplitude at slow speed as he ties off a knot and looks up. The weathered old man with a low, gravelly voice and an unhurried pace (S1) says: <d>[English] The tide waits for no one.</d> [Shot 2] At 00:03.500, the camera cuts to a close-up of his calloused hands pulling the knot tight while gulls wheel over the water behind him.

overall_soundscape: Gentle waves lap against the harbour wall and wooden hulls knock softly together. Gulls cry in the distance, and the nylon cord creaks as it is pulled tight.

non_diegetic_music: A solo accordion at a slow tempo plays a simple melody that fades out at the end.
PROMPT

# jq puts the multi-line prompt into the JSON body with its newlines escaped
curl -s https://api.vultrinference.com/v1/videos \
  -H "Authorization: Bearer $VULTR_INFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg prompt "$prompt" '{model: "minimax-h3", prompt: $prompt, duration: 6}')"

import os

from openai import OpenAI

client = OpenAI(base_url="https://api.vultrinference.com/v1", api_key=os.environ["VULTR_INFERENCE_API_KEY"])

# MiniMax-H3's layout: the shots with their dialogue, then the ambient sound, then the score
prompt = """integrated_multimodal_description: [Shot 1] Live-action, cinematic, a medium shot frames an elderly fisherman in a knitted flat cap and an olive jacket sitting on a stone harbour wall at dawn, a green net across his knees and small wooden boats rocking in the water behind him. The camera pushes in with small amplitude at slow speed as he ties off a knot and looks up. The weathered old man with a low, gravelly voice and an unhurried pace (S1) says: <d>[English] The tide waits for no one.</d> [Shot 2] At 00:03.500, the camera cuts to a close-up of his calloused hands pulling the knot tight while gulls wheel over the water behind him.

overall_soundscape: Gentle waves lap against the harbour wall and wooden hulls knock softly together. Gulls cry in the distance, and the nylon cord creaks as it is pulled tight.

non_diegetic_music: A solo accordion at a slow tempo plays a simple melody that fades out at the end."""

video = client.videos.create_and_poll(model="minimax-h3", prompt=prompt, seconds="6")
print(video.status)  # completed, or failed with video.error

if video.status == "completed":
    client.videos.download_content(video.id).write_to_file("fisherman.mp4")

In our test render the line was spoken word for word and the cut landed at 3.58 seconds. The rest of this page is about what to put in the prompt.

The mode follows from what you send#

There is no mode field: the inputs you send choose it, and the first line of the prompt changes with it.

Send frames or references, not both. If a request has both, the frames are used and the references are ignored.

The three fields#

Text to video and the three frame modes all use the same three fields, in this order, separated by a blank line:

integrated_multimodal_description: [Shot 1] ...

overall_soundscape: ...

non_diegetic_music: ...

Keep each sound in its place. Dialogue and singing go only in the description. Music the characters can hear, such as a radio, a busker or someone singing, is part of the scene and goes in the description, not in non_diegetic_music. Use N/A in overall_soundscape only when you want silence. For a video with no sound at all, send generate_audio: false.

Shots and cuts#

Start [Shot 1] with the style and the opening composition, such as Live-action, cinematic, a medium shot frames .... Other styles MiniMax names are 2D-animated, 3D CG, claymation, watercolor and vintage film. The first shot has no time. Each later shot starts with the moment it cuts in:

[Shot 2] At 00:03.500, the camera cuts to a close-up of his hands ...

The times are MM:SS.mmm, must increase from shot to shot, and must fall inside the video (its real length is a little longer than duration). Say the camera cuts to, the shot cuts to or the shot changes to. Use a dissolve, fade or wipe only when you want one. Cut when the shot shows something new: another subject, place, viewpoint or moment. To move closer or reframe, move the camera instead.

Camera moves#

Write a camera move as a sentence inside the shot, from a move, an optional amplitude and an optional speed:

The camera pushes in with small amplitude at slow speed toward the letter in her hands.

The amplitudes are with small amplitude and with large amplitude, and the speeds at slow speed and at fast speed. Leave them out for a medium move at normal speed.

Speech and on-screen text#

Give each person who speaks or sings an ID, (S1), (S2) and so on, and keep it from shot to shot. Where a voice is first heard, describe it: who it is, their age, pitch, timbre, pace or accent. Put the spoken words alone inside <d>, after a language tag:

The weathered old man with a low, gravelly voice and an unhurried pace (S1) says: <d>[English] The tide waits for no one.</d>

• Only the words go inside <d>, exactly as they should be said, in their own language: <d>[Japanese] ...</d> for Japanese. Write the rest of the prompt in English.
• Two voices at once are (S1,S2).
• A voiceover is says in an off-screen voiceover:, followed after the <d> by while his lips remain completely closed. Without it, the character on screen may mouth the words.
• A line that runs across a cut has <scenetrans> where the cut splits it, in both shots, and says that the speech continues seamlessly across the cut. A line the end of the video cuts off ends with <cutoff>.
• Text seen on screen, such as a sign, label or subtitle, goes in double quotes, in its own language: a red neon sign reading "营业中".

The video runs a little longer than duration#

H3 renders a whole number of frame blocks at 24 frames a second, so the video is slightly longer than the duration you ask for:

Use the second row for the last frame's time and to plan cuts near the end. A 6-second request has 6.58 seconds of room.

Start from a picture#

Each frame mode begins with one line that says where the pictures sit in the video, then a blank line, then the three fields. Copy the line exactly, dash included, changing only the shot number and the time. The time has two decimals and is the video's real length. The API sends your pictures to the model labelled Picture 1 and, with two frames, Picture 2, so the line and the description refer to them by those names.

A first frame#

Send the image as input_reference, or as frame_images with "frame_type": "first_frame":

For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced.

Open [Shot 1] on what the picture shows: its style, people, composition and setting. Then describe what happens next. Keep the people's faces, clothes and colours as the picture has them. In our test, a rainy street at night opened on the picture and the camera tracked forward behind its cyclist as the description said.

A first and a last frame#

"frame_images": [
  {"type": "image_url", "frame_type": "first_frame", "image_url": {"url": "https://example.com/first.png"}},
  {"type": "image_url", "frame_type": "last_frame", "image_url": {"url": "https://example.com/last.png"}}
]

How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 6.58-second mark of the target video.

Describe the way from one picture to the other: how the subject moves, how the pose changes, how the framing and light shift. Don't describe the two pictures again. Keep it to one shot unless you need a cut; with cuts, Picture 2 is (from Shot N) of the last shot. In our test at duration 6, written with 6.58, the video opened on the first picture and ended on the second.

Only a last frame#

How the reference pictures align with the target video — <Picture 1> (from [Shot 1]) aligns with the 6.58-second mark of the target video.

The picture is where the video ends, so invent where it starts: a plausible earlier moment that leads there. Then describe how everything converges on the picture by the end. In our test, a portrait close-up became the end of a shot that began with the man walking down a quay toward the camera.

References: images, videos and audio#

input_references takes up to 9 images, 3 videos and 3 audio clips, 12 in all. Audio needs at least one image or video beside it. start_time_seconds sets where each reference video starts being read.

The API labels your references for the model by kind, numbered in the order you send them. The first image is <Picture 1>, the second image <Picture 2>, the first video <Video 1>, and the first audio clip <Audio 1>, wherever they sit in the list. Refer to them by these labels.

A reference prompt has six sections, each a heading on its own line:

The task types, joined with + when there are several:

The ways a reference is kept, for retention_analysis, are fully_preserved, partially_preserved, attribute_transfer (its look moves to another subject) and weak_reference (only a likeness). For audio they are fully_copy, partially_copy, reference and weak_reference.

This prompt, sent with a portrait as the first image and a street as the second, rendered the man from the portrait walking down that street, with a cut at 3.04 seconds and his line spoken word for word:

subject_definitions:
<Subject 1> is the elderly man in <Picture 1>, with a weathered, deeply lined face, a grey stubble beard, a knitted flat cap and an olive jacket.
<Subject 2> is the night shopping street in <Picture 2>, with glowing Japanese shop signs, a noodle stall on the right, and wet, reflective pavement.

summary:
[reference generation] The target video shows <Subject 1> walking through <Subject 2> in the rain at night, stopping at the noodle stall and speaking to the cook.

retention_analysis:
<Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - his face, beard, flat cap and olive jacket are retained.
<Subject 2> (appears in [Shot 1], [Shot 2]): fully_preserved - the shop signs, noodle stall and wet pavement are retained.

detailed_description:
The target video is live-action and cinematic, lit by neon signs at night with rain falling.
[Shot 1] A medium-wide shot looks down <Subject 2>, its glowing shop signs reflected on the wet pavement. <Subject 1>, the elderly man in the knitted flat cap and olive jacket, walks toward the camera without an umbrella, rain beading on his cap. The camera tracks backward at slow speed in front of him as he passes the shopfronts and turns toward the steaming noodle stall on the right. [Shot 2] At 00:03.000, the shot cuts to a medium close-up of <Subject 1> under the stall's awning, steam rising past his face. <Subject 1> (S1), in a low, gravelly voice with an unhurried pace, says to the unseen cook: <d>[English] One bowl, and make it hot.</d> He wipes the rain from his cheek and smiles.

overall_soundscape: Rain drums on the awning and hisses on the pavement while a broth pot bubbles at the stall. Footsteps splash through shallow puddles.

non_diegetic_music: N/A

When a person from a reference speaks, give both labels: <Subject 1> (S1). An audio clip used for a voice is defined as <Audio 1> is the voice-timbre reference for <Subject 1> (S1).. Only the voice is followed, not the clip's words: write the words to say in <d>.

Let an agent write them#

The h3-prompt-writing folder is also an agent skill: a coding agent that reads it can turn a short request into a prompt in this layout, with the right first line for your inputs. MiniMax's two references in it, base-en.txt for the three fields and ref-en.txt for references, have complete examples of every mode. When you use them with this API, use the real video length for times, and the labels in the order you send your references.

Quality, length and resolution#

quality picks how many steps H3 renders: turbo (the default), standard or max, each slower and priced higher. resolution: "2K" renders the 768p video and upscales it. duration is 4 to 15 seconds. The model page has the prices.

When something looks wrong#