Temperature Sampling in Autoregressive TTS

Autoregressive text-to-speech (TTS) models synthesize natural speech by predicting acoustic tokens sequentially, but their output quality hinges heavily on decoding parameters. This article explores why temperature sampling tuning is essential to eliminate common audio defects like unnatural stuttering, repetitive loops, and sudden vocal instability. Understanding how temperature alters token probability distributions allows developers to strike the ideal balance between human-like prosodic expressiveness and stable acoustic generation.

The Autoregressive Generation Process

Modern neural TTS architectures—such as VALL-E, Tortoise, and Bark—operate autoregressively. Rather than generating an entire spectrogram at once, they predict speech frame-by-frame or token-by-token, conditioning each new step on previously generated audio and text representations.

At each generation step, the model outputs logits over a discrete codebook of acoustic tokens. These logits are converted into a probability distribution via the softmax function:

\[P(x_i) = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}\]

Here, \(T\) represents the sampling temperature, a hyperparameter that directly scales the sharpness or flatness of the probability distribution.

The Danger of High Temperature: Voice Instability and Noise

When the temperature is set too high (\(T > 1.0\)), the probability distribution across potential acoustic tokens flattens. This forces the model to treat low-probability, outlier tokens as viable options.

In speech synthesis, acoustic tokens dictate minute nuances such as pitch, formant frequencies, voicing, and background noise. Introducing improbable tokens introduces chaotic variability into the acoustic stream. This manifests as:

• Voice Cracking and Pitch Jumps: Rapid, unnatural fluctuations in fundamental frequency (\(F_0\)).
• Acoustic Hallucinations: Unintended background artifacts, whispering, gasping, or static bursts.
• Loss of Speaker Identity: Sudden shifts in timbre or accents mid-sentence because the model selects tokens uncharacteristic of the reference speaker.

Because the system is autoregressive, once a corrupted token is chosen, it enters the context window. The model then attempts to condition subsequent tokens on this corrupted sound, frequently causing the entire generation to derail into unintelligible noise.

The Danger of Low Temperature: Repetition and Stuttering

Conversely, setting the temperature too low (\(T < 0.5\)) or relying purely on greedy search (\(T \to 0\)) sharpens the distribution, forcing the model to pick only the single most probable token at every step.

While this ensures local acoustic stability, it introduces a severe vulnerability to degenerate loops:

• Phonetic Stuttering: If a phoneme’s tail-end tokens have high self-similarity probabilities, greedy or low-temperature generation gets trapped in a local attractor. The model repeatedly generates identical or cyclical phonetic codes, manifesting as endless stuttering (e.g., "b-b-b-b-because").
• Audio Smearing and Drone: Without slight variations, vowel sounds stretch indefinitely into robotic, flat tones.
• Silence Collapse: Models often get stuck in repeating silence or breathing tokens, failing to proceed to the next word in the text sequence.

Achieving Acoustic Stability: The Golden Window

Tuning temperature prevents these dual failure modes by locating the optimal zone where speech sounds dynamic yet cohesive. For most autoregressive audio models, this optimal range typically falls between \(0.6\) and \(0.85\).

In this window:

1. Entropy is Controlled: Improbable acoustic artifacts are suppressed, preserving speaker consistency and phoneme clarity.
2. Diversity is Preserved: Enough variance remains to avoid deterministic repetition traps and robotic monotony, allowing natural pauses, cadence variations, and expressive pitch contours.

Tuning Alongside Complementary Strategies

Temperature tuning should rarely occur in isolation. To maximize stability against stuttering and voice breakdown, temperature is best combined with complementary sampling constraints:

• Top-\(p\) (Nucleus) Sampling: Restricts candidate tokens to the smallest cumulative probability mass (e.g., \(p = 0.85\)–\(0.95\)), filtering out the long tail of erratic tokens even if the temperature is slightly elevated.
• Repetition Penalties: Explicitly penalizes recently generated acoustic tokens to actively break stutter cycles without requiring the temperature to be raised to unstable levels.
• Top-\(k\) Truncation: Caps the choice pool strictly to the top \(k\) candidates, ensuring bounded randomness at every time step.

Careful calibration of temperature alongside these mechanisms is the fundamental prerequisite for delivering clean, hallucination-free, and natural voice synthesis in autoregressive TTS pipelines.