FastSpeech 2 Variance Adaptor: Pitch, Energy, Duration

FastSpeech 2 revolutionized non-autoregressive text-to-speech (TTS) synthesis by introducing the variance adaptor, a core architectural component designed to address the one-to-many mapping problem in speech generation. This article breaks down the role of the variance adaptor, detailing how it models and predicts duration, pitch, and energy directly from phoneme representations to produce expressive, high-fidelity synthetic speech while giving users granular prosodic control.

Resolving the One-to-Many Mapping Problem

In speech synthesis, a single sequence of text can be spoken in countless ways depending on speaking rate, pitch contours, and vocal intensity. When a model relies solely on text input to generate acoustic features, it faces a severe one-to-many mapping problem, often leading to over-smoothed, robotic mel-spectrograms.

The variance adaptor in FastSpeech 2 resolves this by explicitly injecting acoustic variance information—duration, pitch, and energy—into the phoneme representations before they reach the mel-spectrogram decoder. Instead of relying on a complex teacher-student distillation process like its predecessor, FastSpeech 2 extracts these variance features directly from ground-truth speech during training, dramatically simplifying training and improving voice naturalness.

Key Components of the Variance Adaptor

The variance adaptor is positioned between the phoneme encoder and the mel-spectrogram decoder. It sequentially processes phoneme hidden states through three dedicated predictor networks:

1. Duration Predictor and Length Regulator

The duration predictor determines the exact number of acoustic frames each phoneme should occupy.

• Training: Ground-truth phoneme durations are extracted using Montreal Forced Alignment (MFA). The duration predictor is trained using mean squared error (MSE) loss against these targets.
• Length Regulation: Once durations are determined (from ground truth during training or predicted during inference), the length regulator expands the phoneme hidden states according to their predicted duration values. This expands the sequence length from phoneme-level to frame-level, aligning it with the target mel-spectrogram temporal resolution.

2. Pitch Predictor

Pitch conveys intonation, emotion, and sentence structure.

• Extraction: FastSpeech 2 extracts the continuous fundamental frequency (\(F_0\)) contour from the audio signal using tools like PyWorld.
• Embedding: To handle the high dynamic range of pitch, the continuous values are quantized into discrete bins (typically 256 bins on an equidistant scale) and transformed into pitch embeddings.
• Integration: The pitch predictor predicts the \(F_0\) contour directly from the length-regulated hidden states, and the corresponding pitch embedding is added element-wise to the representations.

3. Energy Predictor

Energy corresponds to vocal loudness and intensity.

• Extraction: Energy is computed by calculating the L2-norm of the short-time Fourier transform (STFT) magnitude for each speech frame.
• Embedding: Similar to pitch, the energy values are quantized into discrete bins and mapped to energy embeddings.
• Integration: The energy predictor estimates the frame-level energy, and the resulting embedding is added directly to the hidden states, reinforcing the amplitude profile of the generated speech.

Enabling Prosodic Control and Voice Manipulation

Beyond improving voice naturalness and training stability, the variance adaptor decouples acoustic attributes, enabling fine-grained manipulation at inference time:

• Speaking Rate: Multiplying the predicted duration values by a scaling factor speeds up or slows down the overall speech rate, or alters specific phoneme lengths without altering pitch.
• Pitch Shifting and Dynamic Range: Modifying the predicted pitch contour allows users to transpose the voice higher or lower, flatten it for a monotonous tone, or exaggerate contour peaks for higher emotional expressiveness.
• Volume and Stress Control: Adjusting the predicted energy values allows direct control over speech loudness and emphasis on specific words or syllables.

By explicitly providing these variance attributes as conditional inputs, the variance adaptor removes ambiguity from the acoustic feature generation process, enabling FastSpeech 2 to synthesize fast, stable, and highly expressive speech.