Why Tacotron Skips and Repeats Words in TTS

Tacotron and Tacotron 2 revolutionized neural text-to-speech (TTS) by introducing sequence-to-sequence architectures that generated high-quality mel-spectrograms directly from text. However, their reliance on dynamic, autoregressive attention mechanisms introduced significant stability issues, notably word skipping, word repeating, and catastrophic alignment collapse. This article breaks down the primary architectural and algorithmic causes behind these attention failures in Tacotron-based systems and explains why these models struggled to maintain consistent alignment.

1. Lack of Strict Monotonic Inductive Bias

Human speech is strictly monotonic: words are spoken sequentially from left to right, and a speaker does not jump backward or forward unpredictably. Standard sequence-to-sequence models were originally designed for machine translation, where word reordering is necessary.

Tacotron adopted content-based and additive soft attention (such as Bahdanau attention). Because this mechanism computes a softmax distribution over the entire input text at every decoding step, the model mathematically possesses the freedom to look anywhere in the sequence. While Tacotron 2 introduced location-sensitive attention—incorporating cumulative alignment history to encourage forward movement—it remained a soft constraint rather than a hard physical restriction. When the attention weights diffuse, the focus can easily jump forward (skipping words) or drift backward (repeating words).

2. Exposure Bias from Teacher Forcing

Tacotron models are trained using "teacher forcing," where the decoder is fed the ground-truth acoustic frame from the previous time step. During inference, however, the model must feed its own previously generated (and potentially imperfect) frame back into the decoder.

This discrepancy creates exposure bias. If the decoder generates an acoustic frame with minor artifacts or noise, this error feeds into the next step. The attention mechanism, conditioned on this flawed recurrent state, fails to resolve the correct text position. A small tracking error can cascade into complete alignment loss, causing the attention weights to stall on a single phoneme (leading to repeated words or babbling loops) or skip past unvoiced phonemes entirely.

3. Acoustic Ambiguities and Punctuation Pauses

Text-to-speech alignment struggles when there is no distinct acoustic information corresponding to an input character. Long pauses, silence, and punctuation marks do not generate distinct harmonic structures.

During silent frames, the decoder receives minimal acoustic feedback to distinguish its current position. In these silent gaps, soft attention often distributes weight across multiple parts of the text or flattens out entirely. Once the silence concludes, the model must "guess" where to resume alignment, often landing on the wrong word or re-reading the preceding phrase.

4. Similar Phonetic Contexts

When sentences contain repeated words, identical syllables, or similar phonetic phrases in close proximity (e.g., "she sells sea shells"), the attention mechanism's content-based component receives nearly identical key-value representations. Without an absolute positional guarantee, the location-sensitive filters can be overridden by strong content similarities, causing the alignment matrix to loop between duplicate words or skip the intermediate phrase entirely.

5. Flat Attention Distributions and Premature Termination

Tacotron uses a separate stop token projection to decide when an utterance is complete. If the attention distribution becomes diffuse—spreading evenly across the entire text instead of focusing sharply on a specific phoneme—the decoder's hidden state degrades. This often causes the stop-token classifier to trigger prematurely, cutting the sentence off halfway, or fails to trigger at all, producing continuous static and repeated tail syllables.

These inherent vulnerabilities in dynamic attention ultimately drove the speech synthesis field away from autoregressive attention models, leading to the development of explicit duration predictors and non-autoregressive architectures like FastSpeech and VITS.