Acoustic Features of Whispered Speech in TTS Synthesis

In controllable Text-to-Speech (TTS) synthesis, generating natural whispered speech requires modifying key acoustic parameters that distinguish it from standard modal (voiced) speech. Because whispering eliminates vocal cord vibration, it alters the spectral structure, energy distribution, and temporal dynamics of the vocal signal. This article examines the primary acoustic differences—including the absence of fundamental frequency, formant frequency shifts, altered spectral tilt, high aperiodicity, and duration changes—and their direct implications for controllable speech synthesis.

Absence of Fundamental Frequency (\(F_0\))

The most defining acoustic distinction is the complete absence of vocal fold oscillation. In voiced speech, periodic vocal fold vibration creates a distinct fundamental frequency (\(F_0\)) and a rich harmonic structure. In whispered speech, the glottis remains partially abducted, producing turbulent airflow instead of periodic pulses.

For controllable TTS architectures, this means:

• The \(F_0\) contour must be set to unvoiced (or zeroed out) across all phonemes, including vowels and voiced consonants.
• Pitch perception in whisper is conveyed through vocal tract resonances and formant transitions rather than a physical base frequency.
• Pitch predictors and conditioning modules must be bypassed or adapted to prevent the vocoder from generating buzzing, pseudo-harmonic artifacts.

Formant Shifts and Bandwidth Broadening

The open configuration of the vocal tract during whispering alters acoustic resonances, resulting in distinct changes to formant frequencies and their bandwidths:

• Elevated \(F_1\): The first formant (\(F_1\)) shows the most substantial change, typically shifting upward by 150 to 300 Hz. This is caused by the decoupling of the subglottal and supraglottal cavities through the permanently open glottal chink.
• Shifts in \(F_2\) and \(F_3\): Higher formants exhibit context-dependent upward shifts, altering vowel quality and vowel space compactness.
• Broadened Formant Bandwidths: Increased acoustic losses at the glottis lead to broader, less distinct formant peaks compared to the sharp, narrow resonances found in voiced speech.

Controllable TTS systems must adjust acoustic feature representations (such as mel-spectrogram targets) to reflect these higher, wider formant profiles to prevent synthesized whispers from sounding like low-pass-filtered voiced speech.

Low Harmonic-to-Noise Ratio (HNR) and Aperiodicity

Voiced speech exhibits a high Harmonic-to-Noise Ratio (HNR) due to the dominance of periodic glottal pulses. Whispering replaces this periodic source with continuous, turbulent friction noise generated at the glottal constriction.

• Whispered speech approaches a zero HNR, functioning almost entirely as aperiodic, colored noise.
• Neural vocoders (such as HiFi-GAN or BigVGAN) must switch their source excitation from periodic pitch-synchronous pulses to pure white or shaped noise to render realistic whisper texture.

Spectral Tilt and Energy Redistribution

The spectral slope of whispered speech differs markedly from modal phonation:

• Flatter Spectral Tilt: Voiced speech typically demonstrates a steep spectral falloff (approximately -12 dB per octave) due to the glottal flow wave. Whispered speech features a much flatter spectral tilt, retaining significantly more energy in the high-frequency regions (above 4 kHz).
• Lower Overall Intensity: The root-mean-square (RMS) energy of whispered speech is generally 10 to 20 dB lower than that of voiced speech.

In TTS pipelines, explicit energy scaling and spectral weighting controls are necessary so the model does not attempt to normalize the loudness to match standard conversational levels, which ruins the perceptual intimacy of a whisper.

Temporal and Durational Alterations

Whispered speech exhibits distinct temporal differences compared to voiced speech:

• Vowel Lengthening: Because unvoiced vowels carry lower acoustic energy and degraded spectral cues, speakers naturally compensate by prolonging vowel durations to preserve intelligibility.
• Consonant-to-Vowel Ratio: Unvoiced fricatives and stops naturally dominate the energy envelope, altering the relative timing structure between consonants and vowels.

Controllable duration predictors in modern non-autoregressive TTS models must account for these altered phoneme alignments to ensure the synthesized whisper maintains natural rhythm and phrasing.