Unsupervised Phone-Level Prosody Prediction in TTS

Training fine-grained phone-level prosody predictors without manual annotations is accomplished by combining self-supervised acoustic extraction, latent variable modeling, and automated alignment tools. Modern Text-to-Speech (TTS) frameworks eliminate the need for human labeling by deriving pseudo-ground-truth features directly from raw speech audio through digital signal processing, forced aligners, and pre-trained speech representation models. These extracted features supervise neural prosody predictors during training, allowing models to generate expressive, natural-sounding pitch, energy, and rhythm directly from text input at inference time.

Automated Acoustic Extraction and Forced Alignment

The foundational step in label-free prosody modeling relies on automated feature extraction pipelines. Instead of human annotators marking stress or rhythm, algorithms compute deterministic acoustic metrics directly from speech waveforms:

• Duration: Hidden Markov Model (HMM)-based forced aligners, Montreal Forced Aligner (MFA), or dynamic time warping (DTW) map text phonemes to corresponding temporal boundaries in the audio.
• Fundamental Frequency (\(F_0\)): Pitch tracking algorithms (such as YIN, pYIN, or harvest) extract frame-level pitch contours, which are subsequently aggregated across the aligned phoneme boundaries to produce phone-level pitch averages or contours.
• Energy: The root mean square (RMS) amplitude or short-term frame energy is computed across the waveform and averaged over phone intervals.

These continuous, phone-level targets serve as direct supervision signals for regression modules, such as the duration, pitch, and energy predictors commonly implemented in non-autoregressive architectures like FastSpeech 2.

Latent Representation via Variational Autoencoders (VAEs)

To capture nuanced prosodic elements beyond basic pitch and energy—such as voice quality, micro-intonations, and localized emphasis—reference encoders with information bottlenecks are utilized. During training, a reference speech encoder extracts a high-dimensional continuous representation from the target audio.

By applying phone-level temporal pooling or cross-attention with the phoneme sequence, the model projects the audio representation to the phone domain. A Variational Autoencoder (VAE) structure introduces a Gaussian prior, forcing the latent space to remain smooth and compact. A separate prosody predictor is then trained to predict these latent prosodic vectors conditioned solely on text encodings, ensuring the model avoids relying on target audio during inference.

Discretization with Vector Quantization

Continuous prosodic targets often suffer from regression-to-the-mean, resulting in flat, overly generalized speech output. Vector Quantization (VQ), as seen in VQ-VAEs, discretizes continuous acoustic spaces into codebooks of learned prosodic tokens.

Frame-level speech features are quantized into discrete indices that represent distinct intonational and rhythmic patterns. A sequence-to-sequence network or an autoregressive language model can then be trained to predict these discrete prosodic tokens directly from input phonemes via cross-entropy loss, entirely bypassing the need for manual categorization.

Self-Supervised Speech Models (SSL)

Pre-trained speech representations from models such as HuBERT, WavLM, or wav2vec 2.0 contain rich, multi-layered phonetic and acoustic hierarchies learned from thousands of hours of unlabeled audio. By extracting embeddings from intermediate layers of these networks, practitioners obtain dense prosodic information that captures speaking style, tempo, and emotion.

Because these representations are aligned temporally with the audio, phoneme alignment boundaries can be used to average-pool the SSL embeddings per phone. These pooled vectors function as comprehensive prosodic representations, supervised via contrastive loss or mean squared error (MSE) against the text-driven prosody predictor.