CNN vs Transformer Backbones in Text-to-Speech

Text-to-Speech (TTS) architectures have transitioned from convolutional neural networks (CNNs), such as the DeepVoice series, to self-attention-based Transformer backbones. While CNNs introduced parallelized training and local acoustic modeling to replace slow recurrent networks, Transformers have surpassed them by capturing global context, improving prosodic naturalness, and scaling effectively with massive speech datasets. This article compares CNN and Transformer backbones across receptive fields, inference speed, alignment handling, and computational efficiency.

Context Modeling and Receptive Fields

CNN-based architectures like DeepVoice 3 rely on stacked 1D convolutions to extract temporal features from text and audio representations. Because standard convolutional kernels operate on fixed local neighborhoods, these models require dilated convolutions to expand their receptive field across longer sequences. While effective for phoneme-level transitions, CNNs struggle to naturally capture sentence-level intonation and global prosodic contours without significantly deepening the network.

In contrast, Transformer backbones use multi-head self-attention mechanisms that connect every input token to every other token in a single layer. This provides an unrestricted receptive field from the start. Transformers can model long-range syntactic and semantic relationships across entire sentences, producing speech with more natural stress, rhythm, and expressive pauses.

Alignment and Monotonicity

Mapping phonemes to spectrogram frames requires robust alignment:

• CNNs with Attention (DeepVoice): Models like DeepVoice 3 implement specialized convolutional attention mechanisms with positional encodings and monotonic alignment constraints to prevent skipping or repeating phonemes. Despite these additions, attention errors still occur over unusually long or short utterances.
• Transformer-Based Alignment: Early autoregressive Transformers (like Transformer TTS) similarly relied on attention alignments but suffered from stability issues. Modern non-autoregressive Transformers (such as FastSpeech or Transformer-based latent diffusion models) decouple duration modeling from acoustic generation, using explicit duration predictors or dynamic programming (like Monotonic Alignment Search) to guarantee stability while retaining self-attention's expressive benefits.

Computational Efficiency and Scalability

CNNs scale linearly (\(O(N)\)) with sequence length, making them computationally predictable and memory-efficient for long sequences. Their feed-forward nature avoids recurrent feedback loops, enabling efficient parallel processing on GPUs during both training and inference. As a result, CNN backbones are often easier to optimize for resource-constrained edge devices.

Transformers scale quadratically (\(O(N^2)\)) with sequence length in standard self-attention, requiring more memory and compute for lengthy inputs. However, Transformers scale far better with model size and training data volume. While CNN-based TTS models plateau in quality as datasets grow into tens of thousands of hours, Transformer models (such as modern zero-shot and voice-cloning models) continuously improve in zero-shot generalization, speaker similarity, and multi-speaker robustness.

Acoustic Detail vs. Global Prosody

CNNs excel at processing fine-grained local textures because convolutional operations inherently emphasize neighboring feature patterns. This makes CNNs highly effective at generating clear, artifact-free acoustic spectrogram details at the phoneme boundary level.

Transformers, while superior at high-level prosody and semantic pacing, often lack the innate local inductive bias of convolutions. For this reason, modern state-of-the-art TTS architectures frequently combine the two approaches: utilizing Transformer layers in the core text and duration modeling stages, followed by convolutional or residual convolutional layers (such as ConvNeXt or HiFi-GAN blocks) for frame-level acoustic and vocoder synthesis.