Causes of Emotion Leakage in Multi-Emotion TTS

Training multi-emotion Text-to-Speech (TTS) systems on imbalanced datasets frequently leads to emotion leakage and style collapse, where the generated speech either inappropriately blends emotional tones or loses expressiveness entirely by defaulting to an average, neutral style. This article examines the core technical mechanisms driving these issues, focusing on representation bias, latent space entanglement, conditioning failures, and variational collapse during model training.

Representation Imbalance and Majority Bias

When a dataset contains significantly more neutral or standard speech than expressive categories, the loss function becomes dominated by the majority class. Modern neural TTS architectures update their weights primarily to minimize reconstruction errors on these prevalent samples. As a result, the model develops an inductive bias toward the majority acoustic features—such as standard pitch trajectories, typical speaking rates, and normalized energy levels. Consequently, minority emotions are pulled toward the distribution of the dominant class, leading directly to style collapse.

Feature Entanglement in the Latent Space

Emotion, speaker identity, and phonetic content share acoustic manifestations like fundamental frequency (\(F_0\)), duration, and spectral tilt. In a balanced dataset, a model learns to disentangle these elements through diverse cross-combinations. In an imbalanced dataset, the model struggles to isolate emotional variance from speaker characteristics and text semantics. This entanglement leads to emotion leakage: attributes unique to one emotion (such as the breathiness of sadness or the high energy of excitement) inadvertently bleed into other styles because the latent representations overlap rather than forming distinct, well-separated clusters.

Semantic and Lexical Bias

TTS models naturally correlate text features with prosodic characteristics. If certain emotions in an imbalanced dataset are only paired with specific phrases, the model over-associates lexical patterns with emotional delivery. When tasked with synthesizing an emotional style using neutral or counter-intuitive text, the model experiences a conflict between the text encoder's semantic priors and the emotion conditioning label. This mismatch often causes the model to default to the lexical prior, resulting in leakage of the dominant semantic tone rather than rendering the intended emotional style.

Posterior Collapse in Latent Variable Models

Many expressive TTS architectures utilize Reference Encoders, Global Style Tokens (GSTs), or Variational Autoencoders (VAEs) to capture fine-grained prosody. In an imbalanced regime, variational models are highly susceptible to posterior collapse. The Kullback-Leibler (KL) divergence term forces the approximate posterior of minority classes to match the standard Gaussian prior, causing the latent variables to become uninformative. When the conditioning variable carries insufficient distinct information, the decoder ignores it, causing style collapse where all emotion labels generate nearly identical outputs.

Weak Conditioning and Label Over-Smoothing

Using simple one-hot emotion embeddings often fails to enforce hard boundaries between expressive states when data is skewed. Because gradient updates for minority classes are sparse, the learned embeddings for rare emotions remain close to their initial values or cluster near the center of the embedding space. This weak separation prevents the acoustic decoder from learning distinct conditional priors, allowing acoustic features to drift across emotional boundaries.