Reading the timings back out of my own narration

From the build of Still Light →

Before there was a game there was the devlog video, and it was the last piece to actually work. The video is easy to describe and annoying to build: a narration track, a stack of frames, and cuts that land on the right words. The narration is synthesized. The frames are screenshots I took while working. The cuts are the problem.

The API knows when it said nothing

Text-to-speech on Atlas Cloud goes through elevenlabs/v3/text-to-speech. Its entire parameter surface is text, voice, stability, and apply_text_normalization. No alignment output, no character timings, no word timings. I hand it a script and it hands me back a wav. That is fine for listening and useless for editing, because to cut a frame at "and then the whole thing crashed" I need to know when it said that.

The safe answer is to never ask. Synthesize each storyboard segment as its own request, measure the wav durations, add them up. That is exact, and it is still the fallback. It is also billed per request at $0.10 a call, so a five-segment episode pays $0.50 to learn what one $0.10 call plus a timer would have told me.

The other answer is funnier. Speech recognition is two orders of magnitude cheaper than speech synthesis. bytedance/seed-asr-2.0 costs $0.002 a call and returns per-word timestamps. So I transcribe my own narration, audio I generated from a script I already have, purely to recover the timing metadata the synthesizer declined to emit. The round trip costs about 2% of the narration it measures, cheap enough that the final render always does it.

The MCP tool ate the payload

Atlas is wired in as an MCP server, and the first version of this went through atlas_get_prediction like everything else. It reported success. It returned no words.

MCP tools are rendered for a model to read. atlas_get_prediction composes a tidy human summary of the prediction (status, a line about the output, a URL) and drops everything it does not consider worth saying out loud, stt_result included. Nothing is broken. The tool is doing its job. Its job just isn't "be a typed API client." The tool result is prose, and prose is a lossy encoding of a payload.

The fix is a split that now feels obviously correct: submit through MCP, poll over REST with the same key.

/**
 * Poll the REST endpoint rather than atlas_get_prediction: the MCP tool renders a
 * human summary and drops the structured payload (stt_result word timings, etc).
 */
const res = await fetch(`${API}/${id}`, { headers: { Authorization: `Bearer ${key}` } })

The prediction id itself has to be scraped out of the submit response, where it arrives as the one backticked hex token in a sentence. That is the shape of the seam: MCP for the thing a model is good at (choosing a model, validating params against a schema before spending credits), HTTP for the thing a program is good at (reading a number out of a field).

The transcript is not the script

With stt_result.words[] in hand (the documented utterances[] shape never materialized, so the parser accepts both) there is one more gap. ASR gives me what it heard. The storyboard knows what was written. They are close, not equal: a word gets dropped, two get merged, punctuation moves.

So the alignment is a monotone greedy match with a short lookahead. Walk the transcript, look for each word in the next twelve script words, and when it hits, advance the cursor:

for (let k = si; k < Math.min(script.length, si + 12); k++)
  if (script[k].n === w.n) {
    found = k
    break
  }

Words that miss get attributed to whatever segment the cursor is sitting in. Then the interesting part, which is not the matching but the distrust: if any segment ends up with zero matched words, or if the segment ranges come out non-monotonic, the whole ASR result is thrown away and the measured durations win. Cuts land on the midpoint of each inter-segment pause, and boundaries are forced contiguous, because a gap of even a few frames makes the assembled video drift short of the audio.

One more trap: the timings arrive in seconds, except when they don't. If the largest timestamp is more than twice the known narration length, they're milliseconds, and everything gets scaled by a thousand.

What I took from it

Two things generalize past this video. First, a missing output is sometimes just a cheaper model away: the inverse operation may be commodity-priced even when the forward one isn't. Second, an MCP tool result is a rendering, not a record. When the bytes matter, go find the endpoint underneath.

And keep the boring path alive. The expensive, clever, word-accurate route is the one I reach for first; the dumb sum of wav durations is the one that means a bad transcript costs me nothing but a slightly less snappy cut.

← All posts