Why Neural Vocoders Struggle with Unvoiced Fricatives

Neural vocoders in modern text-to-speech (TTS) systems have achieved near-human naturalness for voiced speech, yet they consistently encounter difficulties when synthesizing unvoiced fricatives such as /s/, /f/, and /ʃ/. This article examines the core reasons behind this disparity, contrasting the predictable, periodic nature of harmonic vowels with the stochastic, high-frequency turbulence of unvoiced fricatives. It breaks down how differences in acoustic properties, intermediate representations, loss functions, and network architectures combine to make unvoiced fricatives a distinct challenge in speech synthesis.

Acoustic Divergence: Periodicity vs. Stochastic Turbulence

Vowels are voiced sounds produced by the regular vibration of the vocal folds, which creates a rich harmonic spectrum characterized by a distinct fundamental frequency (\(F_0\)) and clear formant structures. Deep learning models excel at learning and reproducing these periodic waveforms because their phase and amplitude relationships follow highly correlated, predictable patterns over time.

In contrast, unvoiced fricatives are produced without vocal fold vibration. Instead, turbulent airflow is forced through a narrow constriction in the vocal tract. Acoustically, these sounds are not periodic; they are stochastic, continuous-spectrum noise. Generating believable noise requires neural networks to output quasi-random, uncorrelated values rather than the structured, sinusoidal trajectories typical of harmonic sounds.

Information Loss in Mel-Spectrograms

Most neural vocoders reconstruct raw audio from mel-spectrograms, an intermediate representation designed around human auditory perception. The mel scale allocates fine frequency resolution to lower bands while compressing higher bands.

• Vowels: Formants typically sit below 4 kHz, where the mel-spectrogram retains high spectral resolution. This allows the vocoder to easily capture fundamental frequencies and lower resonances.
• Fricatives: Acoustic energy for fricatives—especially the sibilants /s/ and /z/—often extends from 4 kHz well above 8 kHz, reaching up to 16 kHz or higher. The heavy compression of the mel scale at these higher frequencies discards subtle spectral cues and shape details, leaving the vocoder with insufficient structural information to synthesize crisp, distinct noise profiles.

The Penalty of Deterministic Loss Functions

Neural vocoders are commonly trained using deterministic loss functions, such as L1 or L2 spectral distances and multi-resolution Short-Time Fourier Transform (STFT) losses.

For harmonic vowels, minimizing spectral differences encourages the model to align fundamental frequencies and formants accurately. For unvoiced fricatives, however, matching exact target waveforms or deterministic phase components is impossible due to their random nature. When forced to minimize error on inherently non-deterministic signals, regression losses cause the model to output the statistical mean of all possible noise patterns. This regression-to-the-mean effect yields blurred, muffled, or low-energy sound, often perceived by listeners as a "lisp," unnatural whispering, or dull articulation.

Architectural Biases Toward Periodicity

State-of-the-art neural vocoders rely on architectures optimized for resonant signals. Many employ dilated convolutions, periodic activation functions, or source-filter decompositions specifically designed to model repeating pitch patterns efficiently.

While these inductive biases enable models to generate pristine harmonic series with low parameter counts, they inherently resist the generation of pure, uncorrelated white or pink noise. When a periodic-biased network attempts to synthesize an unvoiced segment, it often introduces subtle tonal artifacts, buzzing, or metallic timbre into the output, breaking the illusion of natural turbulence.