Stop TTS Hallucinations and Chatter During Pauses

Conversational Text-to-Speech (TTS) models frequently suffer from "chatter" or audio hallucinations—such as phantom whispers, murmurs, or ambient babble—when attempting to render natural pauses between utterances. This guide breaks down why autoregressive and diffusion-based acoustic models generate these artifacts during silence, and outlines the exact engineering, architectural, and post-processing strategies required to enforce clean, natural pauses in conversational pipelines.

Understanding the Cause of Silence Hallucinations

Modern TTS models trained on massive, unstructured datasets (such as podcasts, audiobooks, and live conversations) learn that natural human pauses rarely contain absolute digital silence. Instead, pauses in the training data are filled with room tone, breath sounds, distant background voices, and throat clearing. When an autoregressive model encounters a pause token or an extended duration without explicit phoneme conditioning, its prediction entropy spikes. The model attempts to resolve this uncertainty by generating probabilistic "filler" sounds, resulting in hallucinated speech or background chatter.

1. Explicit Duration Modeling and Tokenization

Relying on implicit silence generation is the primary cause of pause degradation. Models should instead utilize explicit structural controls for silence:

• Dedicated Pause Tokens: Replace variable whitespace or punctuation-based pauses with discrete duration tokens (e.g., <break time="500ms"/> or <silence_0.5s>).
• Non-Autoregressive Duration Predictors: Decouple phoneme duration from audio frame generation. Systems utilizing architectures similar to FastSpeech or explicit alignment networks assign a fixed duration to silence, strictly limiting the number of latent frames allocated to a pause. Once the frame budget is reached, generation immediately shifts to the next phoneme.

2. Dynamic Sampling and Decoding Constraints

During active speech, higher temperature sampling adds realism and expressiveness. However, maintaining high temperature during pauses significantly increases the likelihood of sampling unintended acoustic artifacts.

• Dynamic Temperature Scaling: Automatically lower the sampling temperature (e.g., dropping from 0.8 to 0.1) when generating frames conditioned on silence tokens.
• Logit Masking: For autoregressive models operating in the token domain, suppress the probability of non-silence acoustic tokens during designated pause intervals to prevent the model from pivoting into vocalization.
• Classifier-Free Guidance (CFG): Increase CFG scale values during pauses to force the model to adhere strictly to the non-speech prompt, suppressing the unconditioned ambient sound distribution.

3. Training Data Sanitization and Conditioning

If a model is trained on noisy raw audio, it will reproduce that noise as a feature rather than an error.

• Acoustic Environment Tagging: During pre-processing, annotate training data with signal-to-noise ratio (SNR) scores or acoustic tags (e.g., studio_clean, background_noise, room_reverb). During inference, conditioning the model explicitly on clean_studio prompts forces the model to synthesize true silence rather than ambient noise during breaks.
• Negative Prompting: In flow-matching and diffusion architectures, implement negative prompting during generation to penalize tokens or audio states that match background speech or unintelligible murmuring.

4. Downstream Voice Activity Detection (VAD) and Gating

When low-latency streaming precludes complex inference-time constraints, deterministic audio engineering applied downstream guarantees clean pauses:

• VAD-Driven Gating: Run generated audio chunks through a fast Voice Activity Detector. If the model is processing a pause interval and low-amplitude acoustic energy is detected, an automated noise gate can zero the output.
• Synthesized Room Tone Infilling: Absolute digital silence can sound jarring to human listeners. Instead of letting the neural network generate the silence, strip the generated audio during the pause entirely and replace it with a loop of flat, standardized, non-vocal comfort noise (dithered room tone).