Detecting Dropped Words in TTS Audiobook Datasets

Building high-quality Text-to-Speech (TTS) models requires clean, verbatim alignment between audio recordings and reference transcripts. In massive audiobook corpora, narrators frequently paraphrase, skip small function words, or misread complex sentences. Training on these noisy pairs introduces severe artifacts, including hallucinated speech, audio cutoffs, and unnatural pacing. This article outlines the primary automated techniques—ranging from forced alignment heuristics to ASR-based transcript diffing and confidence metrics—used to detect and filter out dropped words and misread text at scale.

1. ASR Transcription and Levenshtein Diffing

The most direct way to identify discrepancies between an audiobook recording and its script is to transcribe the audio using an independent, high-accuracy Automatic Speech Recognition (ASR) model (such as Whisper or an optimized Conformer model) and compute the difference.

• Word Error Rate (WER) and Character Error Rate (CER) Filtering: Compute the Levenshtein distance between the raw script and the ASR transcript. Sentences or audio chunks exceeding an empirical WER threshold (typically 5% to 10%) are automatically flagged or purged.
• Token-Level Diff Analysis: By inspecting the specific edit operations (Insertions, Deletions, Substitutions), automated pipelines can detect dropped words (deletions in the ASR output) or misread vocabulary (substitutions).
• Normalized Text Comparison: Prior to diffing, both the script and the ASR transcript must pass through identical text normalization pipelines (expanding numbers, standardizing contractions, and stripping punctuation) to ensure the diff highlights genuine narrator errors rather than formatting variations.

2. Forced Alignment Duration and Boundary Checks

Forced aligners map known text to audio at the phoneme or word level by optimizing an acoustic model against the audio wave. The resulting temporal boundaries reveal physical anomalies that signify skipped or mispronounced text.

• Near-Zero Token Durations: When a narrator drops a word, forced alignment algorithms (such as the Montreal Forced Aligner or CTC-based aligners) frequently compress that word into an unnaturally tiny time window (e.g., under 20 milliseconds) to fit the global acoustic path. Detecting phonemes or words below a minimum physical duration threshold automatically identifies skipped segments.
• Outlier Speech Rates: Calculating the speaking rate (characters or syllables per second) for each segment highlights irregularities. An abnormally high syllable rate often indicates the narrator skipped text, while a very low rate indicates inserted words, false starts, or repeated sentences.

3. CTC and Attention-Based Alignment Confidence

Modern alignment architectures generate confidence scores based on how well the acoustic features match the expected phonemic sequence.

• Acoustic Posterior Probabilities: Aligners generate frame-level posterior probabilities for each expected phoneme. When a word is misread or pronounced incorrectly, the acoustic model outputs low probability scores across that specific token. Segments can be rejected if the minimum or average token confidence drops below a predetermined limit.
• Monotonicity Checks in Cross-Attention: Attention-based alignment models typically display a strictly diagonal attention matrix between text tokens and audio frames. A flattened, discontinuous, or skipping attention path signals that the narrator either jumped ahead (dropped words) or repeated text.

4. Semantic Similarity Verification

While ASR diffing flags exact text mismatches, narrators sometimes substitute synonyms or drop non-essential clauses that traditional diffs might over-penalize or fail to interpret contextually.

• Embedding Distance: Encoding the original script sentence and the ASR-transcribed sentence with a sentence-transformer model allows the system to compute cosine similarity.
• Thresholding Major Omissions: A drop in semantic similarity without a high raw edit distance often points to the omission of critical words (like negations such as "not" or "never") that radically change the meaning of the sentence.

5. Multi-Stage Automated Filtering Pipeline

Processing thousands of hours of audio requires balancing computational cost with precision. Production TTS pipelines typically employ a tiered filtering approach:

1. Coarse Audio Splitting: Segment the continuous audiobook into sentence- or paragraph-level chunks using Voice Activity Detection (VAD) and script anchors.
2. Fast Alignment and Duration Filtering: Run a lightweight CTC aligner to reject segments with severe duration anomalies and zero-length tokens.
3. ASR Verification: Run flagged or boundary segments through an ASR model to precisely log deletions and substitutions.
4. Targeted Slicing or Rejection: Discard segments with structural misreads entirely, or slice out cleanly aligned sub-phrases to preserve usable training audio.