Optimizing TTS for Hearing-Impaired Listeners

Standard Text-to-Speech (TTS) systems are typically trained on clear speech intended for normal-hearing individuals, which often lacks the acoustic features necessary for those with auditory deficits. To improve speech intelligibility for hearing-impaired listeners, researchers and engineers modify acoustic models by altering spectral balance, manipulating phoneme durations, enhancing formant contrasts, and integrating hearing-loss simulation directly into the training loss functions. These adjustments ensure that synthesized speech retains critical linguistic cues despite reduced frequency selectivity and temporal resolution in the listener's auditory system.

Spectral Modification and High-Frequency Boosting

Sensorineural hearing loss most commonly degrades high-frequency sensitivity, making it difficult to distinguish voiceless fricatives, stops, and plosives (such as /s/, /t/, and /f/). Acoustic models are adjusted to alter the generated spectral tilt:

• High-Frequency Emphasis: The model is biased during mel-spectrogram generation to allocate greater acoustic energy to frequencies above 2 kHz to 4 kHz.
• Reduction of Upward Spread of Masking: Intense low-frequency vowel sounds can obscure weaker, subsequent high-frequency consonants. Acoustic models are conditioned to attenuate low-frequency energy (below 500 Hz) to prevent masking effects within the impaired cochlea.

Temporal Adjustments and Consonant Lengthening

Impaired temporal processing makes rapid acoustic transitions hard to resolve. TTS duration predictors—such as those used in FastSpeech or VITS architectures—are modified to address this:

• Phoneme-Specific Duration Scaling: Rather than simply slowing down the entire utterance, duration predictors selectively prolong transient sounds, unvoiced consonants, and transition boundaries while leaving stable vowel nuclei relatively unchanged.
• Pause Insertion and Articulation Rate: Acoustic models are tuned to insert slightly longer inter-word pauses and reduce overall speaking rates to give the auditory cortex more processing time, significantly reducing cognitive load.

Formant and Spectral Contrast Enhancement

Reduced frequency selectivity causes the auditory filters of hearing-impaired individuals to broaden, blurring neighboring frequencies and formant peaks.

• Formant Sharpening: Acoustic models are adapted to increase the peak-to-valley ratio of speech formants (particularly F1 and F2).
• Dynamic Spectral Flattening and Peaking: By sharpening the spectral envelope in the generated representations, the phonetic identity of vowels and vowel-consonant transitions remains distinct even after passing through degraded physiological auditory filters.

Integrated Dynamic Range Compression (DRC)

Listeners with sensorineural loss often experience recruitment—an abnormal growth in loudness where soft sounds are inaudible, but loud sounds quickly become uncomfortably loud.

• Rather than applying dynamic range compression as a post-processing step (which can introduce artifacts and noise amplification), modern acoustic models generate speech with a naturally narrower dynamic range.
• Energy predictors are constrained to ensure low-energy phonetic segments are raised above typical hearing thresholds while high-energy peaks are managed to stay below discomfort levels.

Auditory-Informed Loss Functions

Modern deep learning-based TTS systems use loss functions that optimize for human perception rather than purely mathematical convergence (like mean squared error).

• Hearing Aid Speech Perception Index (HASPI): Models can be fine-tuned using reinforcement learning or perceptual loss functions based on metrics like HASPI, directly optimizing the synthesized audio for predicted intelligibility scores rather than standard naturalness (MOS).
• Cochlear Model Integration: Differentiable models of the impaired human ear are placed downstream of the acoustic generator during training. The generator learns to output acoustic features specifically adapted to remain intelligible after undergoing the simulated degradation of outer hair cell damage and auditory filter broadening.