Loss Functions for End-to-End TTS Without Mel-Spectrograms

Fully end-to-end Text-to-Speech (TTS) models bypass traditional intermediate representations, such as mel-spectrograms, to synthesize raw audio waveforms directly from text. Achieving stability and high acoustic fidelity in these architectures requires balancing multiple complementary training objectives. This article outlines the primary loss functions combined in state-of-the-art systems—such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and normalizing flows—to regulate waveform generation, alignment, and acoustic naturalness without mel-scale supervision.

1. Adversarial Loss (GAN Loss)

Adversarial loss guides the generator to produce perceptually realistic waveforms by forcing it to fool dynamic discriminator networks. In end-to-end setups, discriminators typically operate across multiple domains:

The objective is typically formulated using Least Squares GAN (LSGAN) or Hinge loss, which provides smoother and more stable gradients than standard cross-entropy during raw waveform generation.

2. Feature Matching Loss

Direct adversarial loss can cause training instability and mode collapse. To counteract this, feature matching loss acts as an auxiliary regularizer. It calculates the \(L_1\) distance between the intermediate feature representations of the ground-truth audio and the generated audio across all discriminator layers:

\[\mathcal{L}_{\text{fm}} = \mathbb{E} \left[ \sum_{i=1}^{T} \frac{1}{N_i} \| D^{(i)}(x) - D^{(i)}(\hat{x}) \|_1 \right]\]

This forces the generator to match the structural statistics of natural speech at varying receptive fields, significantly stabilizing the early stages of training.

3. Multi-Resolution STFT Loss

Rather than predicting mel-spectrograms as an intermediate target, models apply multi-resolution Short-Time Fourier Transform (STFT) loss directly to the synthesized raw waveform. This loss compares the spectral magnitude and log-magnitude of the generated audio against the target using multiple window lengths, hop sizes, and FFT sizes:

By combining several STFT resolutions simultaneously, the loss balances time-frequency resolution trade-offs without forcing the network into a fixed intermediate acoustic bottleneck.

4. Kullback-Leibler (KL) Divergence Loss

Modern direct text-to-waveform models (such as VITS) often employ Conditional Variational Autoencoders (CVAEs) enhanced with Normalizing Flows. During training, two representations exist:

  1. A posterior encoder that processes the target raw waveform to output a latent distribution.
  2. A prior encoder that processes phonemic text features into a prior distribution.

KL divergence minimizes the statistical distance between the posterior distribution \(q(z|x)\) and the text-conditioned prior distribution \(p(z|c)\):

\[\mathcal{L}_{\text{kl}} = D_{\text{KL}}(q(z|x) \parallel p(z|c))\]

This objective aligns the continuous acoustic space with the discrete text space, enabling inference from text alone without needing ground-truth audio inputs.

5. Duration and Alignment Losses

Without explicit mel-spectrogram frames, establishing temporal correspondence between phonemes and audio samples requires internal alignment mechanisms. Systems frequently use Monotonic Alignment Search (MAS) or dynamic programming to establish optimal path alignments between the prior and posterior representations.

Once this alignment path is determined: