Emotion Transfer in TTS Without Losing Speaker Identity

Cross-speaker emotion transfer in Text-to-Speech (TTS) allows a model to apply the expressive emotional style of a source voice to a different target voice while keeping the target speaker's unique vocal characteristics intact. Achieving this requires separating emotional expressions—such as pitch fluctuations, rhythm, and vocal effort—from timbre and anatomical vocal traits. This article details the primary architectural frameworks and modeling techniques used to decouple emotion from speaker identity to achieve clean, identity-preserving emotional speech synthesis.

Disentangled Latent Representations

The primary technical barrier to cross-speaker transfer is feature entanglement: fundamental frequency (\(F_0\)), energy, and spectral features carry both identity information (vocal tract geometry) and emotional information (arousal and valence).

To solve this, modern TTS architectures utilize separate encoders to extract orthogonal latent spaces:

• Speaker Encoders: Extract a static identity embedding (such as an x-vector or d-vector) that models voice timbre, vocal tract resonances, and time-invariant traits.
• Emotion/Prosody Encoders: Extract dynamic, time-varying embeddings from a reference audio signal to capture style, pacing, and emotional nuance.

Variational Autoencoders (VAEs) and Vector Quantized Variational Autoencoders (VQ-VAEs) are commonly used for the emotion bottleneck. By constraining the information capacity of the emotion latent space, the model learns coarse emotional prosody without memorizing high-frequency spectral details unique to the source speaker.

Adversarial Disentanglement via Gradient Reversal

Even with separate encoders, emotion representations frequently leak speaker-specific traits. Adversarial training prevents this leakage through Gradient Reversal Layers (GRL).

In this setup:

1. An emotion encoder extracts an emotion representation from the reference audio.
2. A speaker discriminator tries to identify the reference speaker from that emotion representation.
3. During backpropagation, the gradients from the speaker discriminator are inverted before reaching the emotion encoder.

This forces the emotion encoder to minimize speaker classification accuracy, systematically stripping identity cues out of the emotion representation and ensuring it contains only style-related acoustic variations.

Pitch and Energy Normalization

Raw prosodic features like pitch contours cannot be directly transferred from one speaker to another, as a reference speaker’s emotional pitch shifts might exceed or distort the physiological pitch range of the target speaker.

Systems address this using speaker-relative acoustic normalization:

• Z-Score Normalization: The model normalizes \(F_0\) and energy sequences against the source speaker's average baseline (\(F_0 - \mu / \sigma\)).
• Target Re-scaling: The normalized deviations are projected onto the target speaker's mean and variance statistics. This enables the target voice to express the relative emotional excitement, depression, or tension of the source voice within its own natural physiological range.

Explicit Prosody Decomposition in Modern Neural TTS

Modern diffusion-based, Flow-Matching, and autoregressive models implement cross-speaker emotion transfer via explicit conditioning:

• Dual Conditioning Modules: The acoustic decoder (e.g., FastSpeech variants, Grad-TTS, or modern Transformer decoders) receives text tokens, the speaker embedding, and the disentangled emotion embedding through separate cross-attention mechanisms.
• Prosody Predictors with Speaker Conditioning: When generating speech purely from text (without a reference audio file at inference), variance adaptors predict duration, pitch, and energy conditioned jointly on the target speaker's ID and an emotion label. This ensures that the generated emotional contours are naturally tailored to the target speaker's unique cadence.

By combining information-constrained bottlenecks, adversarial identity removal, and speaker-relative prosody normalization, TTS systems successfully transfer subtle and extreme emotional states without altering the target speaker's acoustic identity.