How VITS Eliminates Mel-Spectrograms in TTS

Traditional text-to-speech (TTS) pipelines rely on a two-stage process: an acoustic model first converts text into an intermediate acoustic representation (typically a mel-spectrogram), and a separate neural vocoder then synthesizes that representation into a raw audio waveform. VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) eliminates this intermediate mel-spectrogram bottleneck by unifying acoustic modeling and vocoding into a single, fully differentiable framework. By combining Conditional Variational Autoencoders (VAEs), normalizing flows, Monotonic Alignment Search (MAS), and adversarial training, VITS maps text directly to raw waveforms in an end-to-end manner.

The Limitations of the Mel-Spectrogram Bottleneck

Conventional two-stage systems suffer from two primary drawbacks:

1. Error Propagation: Mismatches between the synthetic mel-spectrograms produced by the acoustic model and the ground-truth spectrograms the vocoder was trained on introduce acoustic artifacts and degraded audio quality.
2. Sub-optimal Optimization: Because the acoustic model and vocoder are trained independently with different loss objectives, the system cannot be optimized globally from input text to final waveform.

Latent Variables Instead of Spectrograms

Rather than predicting a deterministic, hand-crafted feature map like a mel-spectrogram, VITS models the connection between text and audio via a continuous latent variable (\(z\)) within a Conditional VAE framework.

During training:

• The Posterior Encoder takes linear spectrograms derived from target speech waveforms and encodes them into a posterior latent distribution \(q(z|x)\).
• The Prior Encoder takes phoneme sequences derived from input text and processes them via a text encoder to produce a prior latent distribution.
• Normalizing Flows are applied to the prior distribution. Because speech possesses complex, multimodal variations in pitch, rhythm, and timbre, a standard isotropic Gaussian prior is insufficient. Normalizing flows apply invertible transformations to the simple prior, reshaping it into a complex distribution that can match the posterior distribution generated by the real audio.

During training, the Kullback-Leibler (KL) divergence loss minimizes the difference between the prior distribution (derived from text) and the posterior distribution (derived from audio). At inference time, the posterior encoder is discarded entirely, and the model samples directly from the transformed text prior.

Alignment via Monotonic Alignment Search (MAS)

Because text tokens and audio frames have different lengths, the model must learn a temporal alignment between them. Traditional systems rely on external aligners or attention mechanisms operating on mel-spectrograms.

VITS addresses this using Monotonic Alignment Search (MAS). MAS calculates the maximum likelihood path between the text representations and the audio latent variables directly in the latent space. This alignment assigns duration targets to each phoneme. A stochastic duration predictor, trained alongside the system via flow-based variational inference, learns to predict realistic speech rhythms and durations directly from text, ensuring natural prosody without external alignment targets.

Direct Waveform Generation with Adversarial Learning

Once the latent representation \(z\) is established, it must be converted into sound. Instead of decoding \(z\) into a spectrogram, VITS uses a decoder based on the HiFi-GAN architecture that operates directly in the time domain.

The decoder takes the latent variable \(z\) and applies a series of transposed convolutions and multi-receptive field fusion modules to upsample the latent representation directly into a 1D raw waveform. The decoder is trained using a combination of:

• Adversarial Loss: Multi-period and multi-scale discriminators evaluate the generated waveforms against real recordings, driving the generator to produce high-frequency details and natural textures.
• Feature Matching Loss: Measures the intermediate feature differences between real and generated audio within the discriminator layers.
• Mel-Spectrogram Loss: Although mel-spectrograms are not generated as an intermediate representation, a reconstruction loss computed on the mel-scale of the final output waveform is used as an auxiliary objective to stabilize adversarial training and ensure tonal accuracy.

The Complete End-to-End Inference Flow

At inference time, the intermediate acoustic representations are completely absent:

1. Input phonemes pass through the text encoder.
2. The stochastic duration predictor determines the timing for each phoneme.
3. The representations are expanded according to these durations and transformed via the inverse normalizing flow into the latent space \(z\).
4. The HiFi-GAN-based decoder translates \(z\) directly into the output waveform.

By optimizing the latent space jointly with the waveform generator, VITS eliminates intermediate representation mismatches, reduces computational overhead during deployment, and achieves natural, high-fidelity speech synthesis directly from text.