Add "Align time codes via forced aligner" to Import plain text - #12833
Open
niksedk wants to merge 9 commits into
Open
Add "Align time codes via forced aligner" to Import plain text#12833niksedk wants to merge 9 commits into
niksedk wants to merge 9 commits into
Conversation
Aligning a script against a full-length video had no good path. The existing "Align via speech-to-text" transcribes first and then matches text to text, which inherits every transcription error; a forced aligner instead matches the script the user already has directly against the audio. crispasr ships this as `--align-only`, and the engine SE already downloads supports it. What it cannot do is take a long file in one pass: the CTC encoder allocates its activations up front and attention is quadratic. Measured on canary-ctc, 60 s of audio aligns in 2 s, 600 s in 68 s and 900 s in 146 s, while a 57 minute file dies asking for a 24 GB buffer. So the work is chunked, which turns out to be faster as well as bounded - 792 s of audio aligns in 56 s chunked against 114 s in a single pass, at the same accuracy. Windows are cut at silence where possible, and each is fed only slightly more script than should fit. Two things the implementation has to get right, both found by testing: - The window slides to wherever the last trusted cue ended, not along a fixed grid. Only ~75% of each window is trusted, so a fixed grid advances the audio faster than the script and the two drift apart over a long file. - The characters-per-second estimate is fixed for the whole file rather than re-learned per window. Learning it feeds back on itself: over-feeding a window makes the aligner compress its cues, a compressed window looks like faster speech, and the next window is over-fed harder still. Overshoot is deliberately small for the same reason. Forced alignment has to consume every token it is given, so once it runs short of audio it takes time from real speech - at 1.7x overshoot alignment held for the first third of a window then drifted by up to 166 s, at 1.15x the distortion stays in the discarded tail. Verified end to end against 13 minutes of speech with per-line ground truth: 34 ms mean start error, 99% of lines within 100 ms, and unchanged at 39 ms when 18% of the script's lines carry wrong or extra words and five spoken lines are missing from the script entirely - which is the point, since the scripts people have are rarely exact transcripts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two bugs made the alignment "way off" on a real script, neither of which my test material could show. Blank lines shifted everything after them. crispasr silently drops blank and whitespace-only rows from its --text-file - feed it five lines with two blanks and three cues come back - while the code matched cue N to line N positionally. Every blank row therefore shifted all following lines by one, cumulatively, for the rest of the file. Only non-blank lines are now sent, and each cue is mapped back through the index it came from. Verified with the real aligner on a script with 77 blank rows among 200 spoken lines: 34 ms mean start error, unchanged from a script with no blanks at all. Cues stretched across silence. A forced aligner ends each segment where the next one begins, so the last line before a stretch of music or action absorbed all of it. On a recording that is 56% speech this produced 40 cues over ten seconds long, one of them 16.8 s, against a true maximum of 4.4 s. Starts are kept as the aligner reports them - they measure within ~34 ms - but durations are now recomputed from reading time and clamped to the configured display limits, the same way the rest of this window does it. Same recording after the fix: longest cue 5.1 s, none over ten seconds, starts unchanged. Both are covered by tests that fail without the fix. The button also went straight to work using whichever aligner speech-to-text happened to be configured with, silently falling back to canary-ctc, with no way to see or change it. It now opens a setup window showing the engine and its version, offering to install or update it, and listing the aligner models with their installed state and download size. The choice is remembered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The window listed install state as plain text ("not installed (442 MB)")
instead of the coloured dot vocabulary used everywhere else. It now uses
StatusDots for both:
- The engine line gets a dot bound to its install state, next to the name
and detected version.
- The aligner combo uses StatusDots.ComboItemTemplate, the same template
the speech-to-text window already uses for this exact list, so each model
shows a dot plus its download size.
- The selected aligner's state is repeated as a dot and line under the
combo, re-checked on selection so a model downloaded while the window is
open shows as installed.
As everywhere else the state is always spelled out in text beside the dot
rather than relying on colour alone, which ComboItemTemplate also surfaces
as a tooltip and accessible name.
Also a waveform glyph on the intro and a cloud-download icon on the engine
button.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A forced aligner cannot skip audio - it must place every line it is given. So when a script omits something that is actually spoken, the lines after the omission get crammed onto the audio of the missing passage and land far too early. Reproduced against ground truth with a contiguous 90 s omission: every line after it was 100-121 s early, mean error 33 s. Scattered omissions are fine - 42 of 200 spoken lines missing still aligns at 34 ms mean - because the aligner absorbs isolated unmatched speech. It is a contiguous passage that breaks it. Squeezed cues are detectable: past the omission, cues ran at 0.36-0.48 of their text's reading time where correctly placed ones ran at 0.84-0.89. Accepting now stops at a sustained run below that ratio, and the audio cursor creeps on without consuming script, so it can walk past the unmatched passage and pick the script up on the far side. This improves the case but does not fix it. Same 90 s omission: every line now gets a time code (was: 85 of 175 dropped) and 74% land within 250 ms, but the worst is still 81 s early, because stepping blindly through audio cannot actually find where the script resumes - that needs to know what is spoken there, which is recognition, not alignment. A clean script is unaffected: 34 ms mean, unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Forced alignment cannot skip audio - it has to place every line it is
given - so a script that leaves out something actually spoken sends the
following lines onto the missing passage's audio. The previous commit's
plausibility check got the worst case from 121 s to 81 s early, but could
not fix it: stepping blindly through audio cannot find where the script
resumes, because that needs to know what is spoken there.
So the transcription does the finding and the aligner does the timing. A
speech recogniser says what is actually spoken and when, ScriptSyncService
matches the script against that transcript - which tolerates text missing
from either side, being a sequence match that may skip - and each run of
confidently matched lines is then forced-aligned against just its own
stretch of audio, which is what gets timings to a few tens of ms.
Measured on 12 minutes of speech with per-line ground truth, script missing
a contiguous run of 25 spoken lines:
mean max within 250 ms
forced aligner 24.02 s 94.83 s 68 %
hybrid 0.04 s 0.10 s 100 %
A clean script is 43 ms mean, so the extra pass costs nothing in accuracy
when the script does match. It does cost a transcription pass, so it is a
checkbox in the setup window - on by default, since a script that matches
the audio exactly is the exception.
Regions are planned between directly-matched lines and never end on an
interpolated one, which would anchor the tail to the wrong place. A region
that fails to align keeps its coarse timing rather than failing the run.
Note the earlier synthetic script was useless for judging this: it was
built from a short cycle of sentence parts, so lines 60 apart were near
identical and no text matcher could tell them apart. The material here uses
distinct vocabulary per line, which is what real dialogue looks like.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The aligner was being handed as much text as a window could hold. That is
exactly what forces it to misplace lines: forced alignment must consume
every line it is given, so a window whose audio contains speech the script
does not cover gets the script crushed onto it. All the machinery on top -
plausibility trimming, cursor creeping, the whole transcription-first
hybrid - was working around a problem the feeding created.
Given slack it does the right thing by itself. Fed 8 lines against a 240 s
window whose matching speech only starts 182 s in, every interior line
landed within 60 ms: it simply emitted blanks over the 182 s it had no text
for. So each window now gets a small chunk - 12 lines, capped at 35% of the
window's worth of reading time - and the rest of the window is left as room
for the aligner to skip whatever the script is silent about.
Two edges of a chunk are not to be trusted, and both are now handled:
- The first cue absorbs all the leading audio the script does not cover, so
its start is meaningless while its end is sound. Its start is taken back
from its end by reading time.
- Once the aligner runs out of script it smears the remaining cues down the
window. Measured: ten good lines, then a cue opening 54 s after its
predecessor closed. Accepting stops at the first cue that opens
implausibly late or lasts far longer than its text takes to read.
Measured on 12 minutes of speech with per-line ground truth, versus the
previous fill-the-window behaviour:
median within 250 ms untimed
clean script 47 ms 97 % 0
25 lines omitted 48 ms 96 % 0
(was, 25 lines omitted) 70 ms 68 % 0
So the missing-text case is now handled by the aligner alone, with no
transcription pass. HybridAligner stays for scripts where even this cannot
find its place, but is no longer the answer to an omission.
The closing lines of a file have no window left to give the aligner slack,
so whatever is still unplaced is laid out by reading time from the last
cue. Approximate, but every line comes out timed.
Removes the fill-the-window planner API this replaces.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The setup window offered "Handle text missing from the script", defaulted it on, and pressing OK therefore opened the speech-to-text window. That was never wanted: the point of a forced aligner is that it does not need a transcription. It was also obsolete. The hybrid existed to work around windows being filled with text, which is what forced lines onto audio they did not belong to. Underfeeding removed that problem at the source - 96% of lines within 250 ms with 25 contiguous lines omitted, no transcription involved - so the extra pass bought nothing but a second model download and double the runtime. Removes HybridAligner, the checkbox, and the SyncResult.DirectMatches field that only the hybrid used. "Align time codes via Speech to text" is untouched; it is a separate button and a separate feature. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first pass has to keep its windows loose - it does not yet know where
anything is - which costs precision at every chunk edge. Once every line
has an approximate position that constraint is gone, so a second pass
re-aligns each batch of 6 lines against a window drawn tightly round where
the first pass put them: a few seconds of audio holding exactly that
batch's speech, which is the condition a forced aligner is best at.
Measured on 12 minutes of speech with per-line ground truth:
median within 100 ms
first pass only 47 ms 94 %
with refine pass 35 ms 96 %
A refined time is only taken when it agrees with the first pass to within
2.5 s. Anchored on a bad first-pass guess the aligner would re-fit a batch
to the wrong audio just as confidently, so disagreement is treated as a
reason to keep what we had.
Progress is now a real percentage on a bar rather than only text, with
aligning filling the first 70% and refining the rest.
Also stops time codes running past the end of the recording - a cue placed
near the end could have reading time added on top and finish after the
video did.
Known limit, unchanged: the closing few lines. The last chunk has no window
left to give the aligner slack, so it can place them late, and the refine
pass will not move them because it cannot tell a late placement from a
correct one. Measured 5 of 200 lines over a second out, worst 17 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The last lines of a file were landing up to 17 s late. The normal loop can
only extend a window forward, and at the end of a recording there is no
forward left, so the final chunk was aligned against just the audio that
remained - the filled-window case again, and it misplaced them exactly as
filling always does.
The last chunk now takes its slack backwards, over a window reaching back
from the end of the recording. Two things had to be right for that to work,
and getting either wrong silently produced the old result:
- Only one line of overlap. Reaching further back drags that speech into
the window as well, which fills it up and removes the very slack this
depends on. A window has to be long *relative to the text in it*, not
just long - rejoining six lines left it exactly full and changed nothing.
- The agreement check must skip the first cue. That cue absorbs the
window's leading audio by design, so comparing its start rejected every
good result and fell through to the deterministic squeeze.
The first cue gets the same leading-absorb correction the main loop uses,
since this window deliberately starts well before the text it holds.
Measured on 12 minutes of speech with per-line ground truth:
median within 250 ms lines over 1 s out
before 35 ms 97 % 5
after 34 ms 100 % 0
The last six lines now land within 60 ms, against 2.9-15.4 s before.
Also hides the subtitle count while aligning - it shares a grid cell with
the progress bar and the two were drawing over each other.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #12832, and the other half of #11746.
"Align via speech-to-text" transcribes first and then matches text to text, so it inherits every transcription error. A forced aligner matches the script the user already has directly against the audio, which is what people actually want when they have a script and a video.
crispasr ships this as
--align-only, and the engine SE already downloads supports it — it was simply never exposed.Why it has to be chunked
The CTC encoder allocates its activations up front and attention is quadratic. Measured on canary-ctc:
failed to allocate buffer, size = 24161.34 MiBChunking isn't a compromise, it's faster and bounded: 792 s of audio aligns in 56 s chunked vs 114 s single-pass, at the same accuracy. The 57 minute file that OOMed completes in 247 s over 20 windows at 1.8 GB peak, which extrapolates to ~8–9 min for a two hour video.
Two things that had to be right
Both of these were found by tests, not by reading:
Overshoot is small (1.15×) for a related reason: forced alignment must consume every token it is given, so once it runs short of audio it takes time from real speech. At 1.7× overshoot alignment held for the first third of a window and then drifted by up to 166 s; at 1.15× the distortion stays in the discarded tail.
Accuracy
Verified end to end through the actual C# path against 13 minutes of speech with per-line ground truth:
Tests
15 unit tests using a fake aligner and fake audio, including a 2-hour case that asserts every line lands within 5 s of where it is actually spoken (this is what catches the drift bug — reintroducing the fixed grid fails it). Plus a self-skipping integration test that drives the real binary; it passes locally at the numbers above and skips in CI, where the 700 MB model isn't available. Full UI suite: 836 passed.
Notes
built-inmeans "let the ASR do its own timing", which standalone alignment can't use, so that falls back to canary-ctc with the usual download prompt.--align-only, which must consume exactly the text it's given.ForcedAlignerOptionstill omits crispasr's FastConformer aligner family (ru, pl, hr, be, fa, ka, hy, uz, kk-ru at ~82 MB each). Worth adding separately — it's the cheapest aligner option and covers languages the wav2vec2 set doesn't reach.🤖 Generated with Claude Code