F0 Extraction and Conditioning in Expressive TTS

In expressive Text-to-Speech (TTS) systems, the fundamental frequency (\(F_0\)) represents speech pitch and serves as a primary vehicle for emotion, prosody, and natural cadence. Training models to accurately capture this dynamic involves extracting \(F_0\) from raw acoustic waveforms, preprocessing it into representations suitable for neural networks, and injecting it into the acoustic or vocoder architecture. This article explains the algorithms used for \(F_0\) extraction, the normalization and modeling techniques applied to pitch contours, and the conditioning pipelines utilized during TTS training to achieve human-like expression.

1. Fundamental Frequency Extraction Techniques

Accurate extraction of \(F_0\) from training corpora is critical, as inaccuracies introduce artifacts or unstable prosody during generation. Two primary paradigms exist for extraction:

• Signal Processing Algorithms: Traditional digital signal processing algorithms analyze periodicity in the time or frequency domain. The pYIN (probabilistic YIN) algorithm uses autocorrelation with hidden Markov models to prune pitch candidates. The Harvest and DIO algorithms (part of the WORLD vocoder framework) extract reliable pitch contours specifically tuned for speech synthesis by evaluating spectral peaks.
• Deep Learning Estimators: Neural pitch trackers, notably CREPE (and its GPU-accelerated implementation TorchCREPE), formulate \(F_0\) extraction as a time-domain classification task over a continuous range of frequencies. Deep learning estimators provide higher robustness against background noise and vocal fry compared to classical autocorrelation methods.

2. Signal Preprocessing and Representation

Raw \(F_0\) values cannot be directly injected into neural networks without standardizing their scale and addressing non-speech regions:

• Voiced/Unvoiced Handling: Unvoiced phonemes (e.g., /s/, /t/) lack vocal fold vibration, resulting in zero or undefined \(F_0\) values. TTS systems typically handle this either by treating the voiced/unvoiced (V/UV) state as a separate binary classification target or by interpolating \(F_0\) linearly across unvoiced regions and training a distinct mask to eliminate pitch during silent or unvoiced segments.
• Log-Scale Conversion: The human perception of pitch is logarithmic rather than linear. Converting raw Hertz values to a logarithmic scale (\(\log(F_0)\)), semitones, or the Mel scale allows acoustic losses to penalize pitch errors in proportion to human auditory perception.
• Normalization: \(F_0\) is standardized per speaker using z-score normalization (\(\mu = 0, \sigma = 1\)) to remove global pitch variance driven strictly by anatomy (e.g., male vs. female vocal tracts). This isolates expressive pitch excursions from speaker identity.

3. F0 Conditioning Pipelines in Training

During training, modern non-autoregressive (e.g., FastSpeech 2, PortaSpeech) and end-to-end (e.g., VITS, StyleTTS) models condition their intermediate representations on \(F_0\).

Teacher-Forcing Phase

During training, the ground-truth \(F_0\) contour is extracted directly from the target audio waveform. This ground-truth contour is passed through a conditioning module (such as a 1D convolutional projection layer or a discrete embedding lookup table if pitch is quantized into bins, typically 256). The resulting pitch embedding matches the temporal resolution of the encoder output—either frame-level or phoneme-level—and is combined with the linguistic representations via element-wise addition or concatenation.

Variance Predictor Training

To enable inference where audio does not exist, an auxiliary module called a Pitch Predictor is trained simultaneously:

1. The text or phoneme encoder outputs hidden representations.
2. The Pitch Predictor (often composed of 1D convolutions, layer normalization, and dropout) predicts the pitch contour from these hidden states.
3. A regression loss (Mean Squared Error or L1 Loss) is applied between the predicted \(F_0\) and the ground-truth \(F_0\).
4. In highly expressive systems, pitch predictors utilize stochastic modeling (such as Variational Autoencoders, Normalizing Flows, or Diffusion modules) to capture multi-modal prosodic variants instead of averaging out the pitch into a flat, robotic contour.

4. Alignment and Resolution Strategies

\(F_0\) conditioning is typically implemented at one of two temporal resolutions:

• Phoneme-Level Conditioning: The pitch contour is averaged across the temporal duration of each phoneme. This abstracts away fine-grained vibrato but focuses the model on macro-prosodic features (such as question inflection or emphasis).
• Frame-Level Conditioning: Pitch is retained at the acoustic frame rate (e.g., every 10 to 12.5 milliseconds, matching the mel-spectrogram hop size). This captures micro-prosodic details, glottal transitions, and expressive intonation. If frame-level conditioning is used, duration predictors align phoneme embeddings to the frame length prior to adding the \(F_0\) vector.

5. Prosodic Control and Synthesis

Conditioning explicitly on \(F_0\) decodes the entanglement between linguistic content, voice timbre, and emotional prosody. Because \(F_0\) enters the acoustic decoder or flow-based vocoder as an explicit parameter, inference-time manipulation becomes possible. Expressive TTS systems can alter mood, emphasize key tokens, or generate questions simply by applying linear scaling, shift transformations, or external reference contours to the extracted \(F_0\) representations before they reach the synthesis layers.