AI video generation API for code, agents and LLMs.

# VlogMe Script Grammar — canonical contract

> Single source of truth for two contracts:
>
> **A. Frontend ↔ Backend (script grammar)** — what humans and AI write.
> **B. Backend ↔ Worker (payload)** — the structured JSON the render fleet consumes.
>
> The frontend never builds a worker payload directly. `flatToPayloads()` in
> `src/lib/scenes/flat-script.ts` is the single bridge between A and B.

---

## A. Script grammar (frontend ↔ backend)

A VlogMe script is plain text driving a short-form render. The project-level
`aspect_ratio` decides the frame (`9:16`, `16:9`, or `1:1`); the script grammar
is identical for all three formats. There are **four explicit beat forms**, plus
a plain-text continuation line for the current avatar. The shape of the token
decides the kind — never the card role.

| #   | Form                   | Meaning                                                                |
| --- | ---------------------- | ---------------------------------------------------------------------- |
| 1   | `@imageN <text>`       | Avatar speaks `<text>`, anchored to photo slot N.                      |
| 2   | `@imageN { <prompt> }` | Standalone video clip generated from photo N. Prompt is in the braces. |
| 3   | `{ @imageM <prompt> }` | Overlay clip on top of the current avatar, using photo M as 1st frame. |
| 4   | `{ <prompt> }`         | Continue — extends the previous clip, no reference image of its own.   |

Plus one media token: `@audioN` on its own line drops audio clip N on the
current avatar.

Plain text on its own line continues speech for the current avatar. This is
mainly used after an overlay beat:

```text
@image1 So I open the laptop.
{ @image3 slow push-in on the glowing keyboard }:3
And the cursor starts moving on its own.
```

### Rules

- Whitespace, blank lines and indentation are **insignificant** — only the
  token shape matters.
- Avatar text (form 1) preserves its newlines verbatim into the worker
  payload's `script` field. Multi-line dialogue stays multi-line end-to-end.
- The current avatar = the most recent avatar speech tag. Forms 3 and 4 anchor
  to that avatar/beat.
- Form 4 requires a preceding rendered frame/current avatar to continue from.
- Form 3 starts directly with `@imageN`: `{ @image3 fade out }`.
- In form 4, the optional keyword `continue` is stripped:
  `{ continue sparks fly }` works the same as `{ sparks fly }`.
- `@imageN-K` (K in 0..100): cross-fade transition code applied when the
  avatar switches (form 1 only).

### Duration override

Any form 2 / 3 / 4 may carry an explicit clip duration in seconds:

- `{ ... }:D` — after the closing brace.
- `@imageN { ... }:D` — same.

Without `:D` the clip uses the system default (3s), capped at 5s.

### Brace body options

Forms 2 / 3 / 4 can use either a plain prompt or an advanced pipe body:

```text
@image2 { slow push-in on a mountain lake }:3
{ @image3 v:glowing OPEN sign, slow push-in | n:watermark, text | s:soft city hum | t5 }:3
{ v:camera keeps drifting upward | am:off }:2
```

Supported segments:

- `v:<visual prompt>` — main visual prompt.
- `n:<negative prompt>` — visual negative prompt.
- `s:<sound prompt>` / `an:<negative sound>` — generated audio prompt fields.
- `@audioM` — uploaded audio clip under this video beat.
- `am:auto|prompt|asset|off` — audio mode.
- `ag:<db>` — audio gain.
- `tK` — transition code, 0..100.

### Examples

```text
@image1 Hey, let me show you something.

@image1 So I open the laptop.
{ @image3 slow push-in on the glowing keyboard }:3
And the cursor starts moving on its own.

@image2 { dolphin breaches in slow motion against the sunrise }:4
{ camera keeps drifting up, sky turns deeper orange }:2

@image1 [shocked] What the hell is happening?
@audio1
```

Breakdown:

- Line 1 — avatar 1 speaks one sentence.
- Line 3 — overlay over the current avatar using photo 3.
- Line 4 — avatar 1 continues speaking.
- Line 6 — standalone video from photo 2 (4s).
- Line 7 — continue clip extending the previous shot (2s, no reference photo).
- Line 9 — avatar 1 again, with an ElevenLabs `[shocked]` audio tag.
- Line 10 — drops audio clip 1 on top of the avatar.

### Legacy back-compat

The old "bare `@imageN` lets the card's role decide" form is still accepted
by the parser. Specifically: an `@imageN` line **without** braces falls back
to `card.role` + `card.effectInsertMode` to pick avatar / video / overlay /
continue. This keeps the timeline editor — which currently edits cards in
the bare form — working unchanged while we migrate the UI to render explicit
brace cards.

When AI generates a script, it should prefer the **four explicit forms**.
Parser/serializer round-trips currently normalize back to the legacy bare form
for the timeline UI; that will change in a follow-up.

### Forbidden

- Mid-line `@imageN` outside of a brace. Each tag is on its own line OR
  inside a `{ … }` block.
- Inline overlay inside avatar speech, e.g. `@image1 words {@image2 ...}:3`.
- Bare `@imageN` video instructions from new AI output. Generate
  `@imageN { ... }:D` instead.
- Nested braces.
- Anything else not listed in §A.

---

## B. Worker payload (backend ↔ worker)

The worker consumes `ScenePayload[]` (defined in `src/lib/scenes/scenes.ts`).
The frontend never builds these — `flatToPayloads()` is the only producer.

Each scene has an `input_type` discriminator:

| `input_type`    | Comes from script form | Key fields                                                                                                                   |
| --------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `text`          | Form 1                 | `image_path`, `script` (avatar speech), `voice_id`, `transition`                                                             |
| `audio`         | `@audioN`              | `image_path` (current avatar), `audio_path`, `audio_duration_seconds`                                                        |
| `ltx`           | Form 2                 | `image_path` (source photo), `script` (visual prompt), `audio_duration_seconds`                                              |
| `video_overlay` | Form 3                 | `image_path` (anchor avatar), `overlay_source_image_path` (photo M), `overlay_visual_prompt`, `overlay_min_duration_seconds` |
| `video_chain`   | Form 4                 | `chain_from_previous: true`, `overlay_source_image_path`, `overlay_visual_prompt`                                            |

Insert variants (overlay / cut) for nested timing inside an avatar audio
chunk follow `docs/public-api-inserts-contract.md`. That doc is the contract
between the frontend builder pipeline and the public render API.

The worker payload shape is **stable** — adding new script grammar features
must NOT require worker changes. Map them in `flatToPayloads()` first.

---

## Pipeline (who owns what)

```text
WishesField                            ← user types brief + uploads media
   │
   ▼
buildScriptFromBrief (server)          ← Gemini generates canonical script
   │
   ▼
useFlatScriptBridge                    ← parses script, mirrors timeline ↔ Code
   │
   ▼
TimelineBlocks                         ← user-editable cards
   │
   ▼
flatToPayloads()                       ← THE bridge: script → ScenePayload[]
   │
   ▼
prepare-render (server)                ← per-scene asset staging, validation
   │
   ▼
render fleet workers                   ← Hunyuan / overlay / avatar
```

**Key principle**: the script grammar is the _only_ thing the user (or AI)
authors. Everything downstream — payload shape, worker inputs, asset paths,
voice IDs, transition codes — is derived. Adding new script behavior =
extend `flatToPayloads()`; don't ask AI or users to write payload JSON.