Location-Sensitive Attention in Tacotron 2

Tacotron 2 achieves stable and natural text-to-speech synthesis by replacing standard content-based attention with location-sensitive attention, solving common failure modes in sequence-to-sequence audio generation. By incorporating historical alignment information through 1D convolutions alongside traditional decoder and encoder states, the mechanism enforces a strictly monotonic reading order. This architectural integration prevents phoneme skipping, word repetition, and alignment drift, enabling the network to reliably generate consistent mel-spectrograms from arbitrary text inputs.

In traditional sequence-to-sequence models, such as machine translation systems, attention is primarily content-based. The decoder determines where to look in the input text by matching its current hidden state against the encoder states. However, text-to-speech generation has a unique constraint: speech is inherently monotonic and strictly ordered from left to right. Content-based attention alone often fails in speech tasks, especially when text contains repeated words, identical phoneme clusters, or long pauses. Under pure content matching, the attention mechanism easily becomes confused, resulting in catastrophic failures such as jumping backward (looping words), jumping forward (skipping phrases), or getting stuck indefinitely.

Tacotron 2 resolves this by implementing a variant of location-sensitive attention, originally introduced by Jan Chorowski and colleagues for speech recognition. Instead of calculating attention weights based solely on the current decoder state and encoder representations, location-sensitive attention takes into account the alignment weights produced in previous time steps.

The integration works by tracking the attention alignment vector from the preceding decoder step. Tacotron 2 passes this prior alignment through a 1D convolution layer with a wide receptive field (typically 32 filters with a kernel size of 31). This convolution extracts positional features from the local context, giving the model an explicit awareness of its recent trajectory across the input text.

Mathematically, the scoring function in Tacotron 2 calculates energy scores by summing three distinct projections:

1. The transformed encoder memory (the textual keys).
2. The transformed decoder hidden state (the acoustic query).
3. The transformed positional features derived from the 1D convolution of the previous alignment.

These combined values are passed through a hyperbolic tangent activation and projected down to a scalar score for each input character. Applying a softmax function over these scores yields the final attention distribution for the current time step.

By adding these convolutional location features, the model gains an inductive bias toward moving progressively forward. The attention mechanism knows where it just read, making it significantly more likely to focus on adjacent phonemes rather than jumping to similar-sounding phonemes elsewhere in the sentence.

This location awareness provides three critical stability improvements to Tacotron 2:

• Mitigation of Repetitions and Skips: The model disambiguates duplicate words (such as "that that" or identical syllables) because the positional context anchors the decoder to the correct instance in the sequence.
• Improved Long-Form Generation: Attention drift is minimized, allowing the network to retain monotonic stability across long sentences where standard attention networks often collapse.
• Clean Termination: Because the attention mechanism tracks its physical progress through the text, the auxiliary "stop token" prediction layer can accurately determine when the final phoneme has been spoken, ensuring the model halts generation promptly without cutting off speech or producing trailing silence.