Pruning Transformer Heads for TTS: Key Trade-Offs

Pruning self-attention heads is an effective strategy for compressing transformer-based Text-to-Speech (TTS) models to achieve faster inference and lower memory footprints on edge devices. However, eliminating these specialized components creates significant trade-offs between computational efficiency and output voice quality. While reducing the number of heads lowers floating-point operations (FLOPs) and inference latency, it directly impacts the model's acoustic fidelity, prosodic expressiveness, phoneme-to-spectrogram alignment stability, and general robustness across diverse linguistic inputs.

Inference Latency vs. Acoustic Fidelity

The most immediate trade-off in head pruning is the balance between hardware execution speed and perceptual audio quality (often measured via Mean Opinion Score, or MOS). Self-attention heads capture diverse representations, from local spectral textures to long-range linguistic context. Pruning redundant or low-magnitude heads yields structured sparsity, translating into direct latency reductions on standard hardware. However, aggressive pruning degrades fine-grained acoustic features, introducing phase anomalies, robotic timbres, metallic buzzing, or muffled formant structures in the generated mel-spectrograms.

Expressive Prosody vs. Parameter Footprint

Different attention heads inherently specialize in linguistic hierarchies, including word stress, pitch contour generation, rhythm, and phrasing. While reducing head count simplifies the model, it frequently flattens expressive variation. Highly pruned models tend to generate speech with monotonous pitch contours and uniform cadence, stripping the synthesized voice of emotional nuance and natural human conversational inflection. For applications requiring strictly functional speech, this trade-off is often acceptable, but for conversational agents, preserving prosodic variability is essential.

Alignment Stability vs. Model Capacity

In autoregressive and non-autoregressive TTS architectures alike, cross-attention and certain self-attention heads are dedicated to learning monotonic, diagonal alignments between text tokens and acoustic frames. If pruning algorithms indiscriminately target or weaken heads responsible for tracking alignment and duration, the model risks catastrophic synthesis failures. These manifest as omitted words, unnatural stuttering, phoneme repetition, and premature cut-offs. Retaining alignment robustness requires constrained pruning strategies or explicit preservation of alignment-critical heads, which inherently limits the maximum achievable compression ratio.

Engineering Savings vs. Fine-Tuning Overhead

Pruning attention heads reduces runtime memory footprint and deployment compute costs, but it increases pipeline complexity during development. Unstructured pruning does not yield practical latency gains on common inference engines, necessitating structured multi-head elimination. Deciding which heads to remove requires compute-intensive sensitivity analysis, such as gradient-based attribution, Taylor expansion importance estimation, or iterative prune-and-retrain cycles. The engineering cost and compute spent during knowledge distillation and fine-tuning must be weighed against the real-world operational savings gained at inference time.

Generalization on Out-of-Domain Inputs

A heavily pruned transformer architecture often retains acceptable performance on standard, in-domain test sentences, but loses its ability to handle complex edge cases. Multi-head diversity provides redundancy that aids in processing rare words, foreign loanwords, unusual punctuation, and domain-specific jargon. Pruning diminishes this phonetic safety net, leading to sharp degradations in intelligibility and pronunciation accuracy when the model encounters inputs outside its primary training distribution.