Reading the timings back out of my own narration

Before there was a game, there was a machine to build the game, and the last piece of that machine to fall into place was the devlog video. The video is easy to describe and annoying to build: a narration track, a stack of frames, and cuts that land on the right words. The narration is generated. The frames are screenshots I took while working. The cuts are the problem.

The API knows when it said nothing

Text-to-speech on Atlas Cloud goes through elevenlabs/v3/text-to-speech. Its entire parameter surface is text, voice, stability, and apply_text_normalization. No alignment output, no character timings, no word timings — you hand it a script and it hands you a wav. That is fine for listening and useless for editing, because to cut a frame at "and then the whole thing crashed" I need to know when it said that.

The safe answer is to never ask: synthesize each storyboard segment as its own request, measure the wav durations, and add up. That is exact, free, and what the pipeline still falls back to. It is also billed per request — $0.10 a call — so a five-segment episode pays $0.50 to learn what one $0.10 call plus a timer would have told me.

The other answer is funnier. Speech recognition is two orders of magnitude cheaper than speech synthesis. bytedance/seed-asr-2.0 costs $0.002 a call and returns per-word timestamps. So I transcribe my own narration — audio I generated, from a script I already have — purely to recover the timing metadata the synthesizer declined to emit. The round trip costs about 2% of the narration it measures, which is cheap enough that the final render always does it.

The MCP tool ate the payload

Atlas is wired in as an MCP server, and the first version of this went through atlas_get_prediction like everything else. It reported success. It returned no words.

MCP tools are rendered for a model to read. atlas_get_prediction composes a tidy human summary of the prediction — status, a line about the output, a URL — and in doing so drops everything it does not consider worth saying out loud, including stt_result. Nothing is broken; the tool is doing its job. Its job just isn't "be a typed API client." The tool result is prose, and prose is a lossy encoding of a payload.

The fix is a split that now feels obviously correct: submit through MCP, poll over REST with the same key.

/**
 * Poll the REST endpoint rather than atlas_get_prediction: the MCP tool renders a
 * human summary and drops the structured payload (stt_result word timings, etc).
 */
const res = await fetch(`${API}/${id}`, { headers: { Authorization: `Bearer ${key}` } })

The prediction id itself has to be scraped out of the submit response, where it arrives as the one backticked hex token in a sentence. That is the shape of the seam: MCP for the thing a model is good at (choosing a model, validating params against a schema before spending credits), HTTP for the thing a program is good at (reading a number out of a field).

The transcript is not the script

With stt_result.words[] in hand — the documented utterances[] shape never materialized, so the parser accepts both — there is one more gap. ASR gives me what it heard. The storyboard knows what was written. They are close, not equal: a word gets dropped, two get merged, punctuation moves.

So the alignment is a monotone greedy match with a short lookahead. Walk the transcript, look for each word in the next twelve script words, and when it hits, advance the cursor:

for (let k = si; k < Math.min(script.length, si + 12); k++)
  if (script[k].n === w.n) {
    found = k
    break
  }

Words that miss get attributed to whatever segment the cursor is sitting in. Then the interesting part, which is not the matching but the distrust: if any segment ends up with zero matched words, or if the segment ranges come out non-monotonic, the whole ASR result is thrown away and the measured durations win. Cuts land on the midpoint of each inter-segment pause, and boundaries are forced contiguous, because a gap of even a few frames makes the assembled video drift short of the audio.

One more trap: the timings arrive in seconds, except when they don't. If the largest timestamp is more than twice the known narration length, they're milliseconds, and everything gets scaled by a thousand.

What I took from it

Two things generalize past this pipeline. First, a missing output is sometimes just a cheaper model away — the inverse operation may be commodity-priced even when the forward one isn't. Second, an MCP tool result is a rendering, not a record. When you need the bytes, go find the endpoint underneath.

And keep the boring path alive. The expensive, clever, word-accurate route is the one the run prefers; the dumb sum of wav durations is the one that means a bad transcript costs me nothing but a slightly less snappy cut.

← All posts