Challenges of Vocal Fry in Personalized TTS

Capturing idiosyncratic vocal qualities like vocal fry and raspy textures represents one of the most complex frontiers in personalized Text-to-Speech (TTS). While modern neural TTS models excel at replicating smooth, modal speech, non-linear voice phenomena present unique mathematical and computational hurdles. This article explores the primary technical challenges involved in synthesizing these rough acoustic features, including the limitations of standard acoustic representations, vocoder artifacts, loss function biases, and training data scarcity in voice-cloning pipelines.

Aperiodic Acoustic Irregularity

Vocal fry (the glottal pulse register) and raspy phonation occur when the vocal folds close incompletely or vibrate irregularly at low frequencies (typically 20 to 50 Hz). This produces strong period-to-period variability known as jitter (frequency variation) and shimmer (amplitude variation).

Traditional acoustic models in TTS rely on predicting continuous, predictable harmonic structures. Because vocal fry relies on deterministic chaos and aperiodic pulses, models struggle to predict the exact timing of these individual glottal bursts. Instead of learning the natural irregularity of a raspy voice, the network often interprets these variations as random noise and suppresses them to minimize error.

Resolution Limits of Mel-Spectrograms

Most modern TTS architectures rely on an intermediate representation, typically a mel-spectrogram, to bridge text processing and audio generation. The mel-scale compresses high-frequency information and averages spectral energy across time-frequency bins.

This compression inherently smooths out the sharp, transient impulses that define a raspy or creaky voice. When a model relies on mel-spectrograms, the subharmonic frequencies and brief silent gaps between distinct vocal fry pulses are blurred together, eliminating the distinct textured "rattle" of the speaker's voice.

Neural Vocoder Failures and Artifacts

Neural vocoders (such as HiFi-GAN, WaveNet, or BigVGAN) are responsible for converting acoustic features back into raw audio waveforms. Most vocoders are trained on large corpora of clean, modal speech where periodic pitch tracking is straightforward.

When presented with non-modal phonation:

• Pitch Trackers Fail: Pitch estimation algorithms often experience octave jumps or fail to track the fundamental frequency (\(F_0\)) during fry, leading to erratic output.
• Phase Incoherence: Vocoders struggle to reconstruct the phase of irregular pulses, causing the synthetic voice to sound robotic, buzzy, or overly hollow rather than naturally raspy.
• Denoising Collapse: Many vocoders mistake high-frequency breathiness (common in raspy voices) for background noise, inadvertently filtering it out.

Objective Function Penalties (Over-Smoothing)

Standard TTS training uses loss functions such as Mean Squared Error (MSE) or L1 loss to compare predicted acoustic features against ground-truth audio. These regression losses statistically penalize extreme outliers and favor the mathematical mean.

Because vocal fry consists of localized bursts of energy punctuated by silence, predicting the exact phase and placement of each pulse is difficult. The model learns that predicting an averaged, continuous, smooth waveform yields a lower mathematical loss than attempting to generate sharp, temporally precise pulses that might be slightly misaligned. This bias inherently strips away idiosyncratic rasp.

Speaker Embedding Dilution in Few-Shot Cloning

Personalized TTS often relies on few-shot adaptation, where a pre-trained base model adapts to a new target speaker using a short reference audio sample. Speaker encoders compress this reference audio into a low-dimensional speaker embedding vector.

These encoders are trained to capture global speaker identity traits—such as general timbre and pitch range—rather than fine-grained, dynamic phonation shifts. As a result, idiosyncratic vocal fry is frequently washed out, leaving the cloned output sounding like a generic, sanitized version of the original voice.