Reduce Accent Contamination in Zero-Shot TTS

Zero-shot multilingual text-to-speech (TTS) models often struggle with foreign accent contamination, where a target speaker's native accent unintentionally bleeds into synthesized speech in another language. This article examines the core technical strategies used to isolate speaker identity from language characteristics. By leveraging feature disentanglement, phonetic normalization, adversarial training, and decoupled prosody modeling, modern TTS architectures can preserve a speaker's unique vocal timbre while rendering foreign languages with native-sounding pronunciation.

Acoustic Feature Disentanglement

Accent leakage primarily occurs when speaker embeddings accidentally capture native phonological patterns alongside physical vocal tract properties. To combat this, systems implement strict disentanglement frameworks:

• Variational Information Bottlenecks: Passing speaker representations through constrained latent spaces (such as VAEs) forces the encoder to discard high-frequency, language-specific articulatory dynamics, retaining only static vocal timbre.
• Vector Quantization (VQ): Quantizing linguistic and speaker latent representations separates continuous speaker characteristics from discrete phonetic tokens, preventing linguistic leakage into the identity vector.

Adversarial Training with Gradient Reversal

Adversarial learning explicitly strips language information from speaker embeddings. During training, a language classifier is tasked with predicting the spoken language directly from the extracted speaker embedding. By inserting a Gradient Reversal Layer (GRL) between the speaker encoder and the language classifier, the encoder is optimized to maximize the language classifier's error. This ensures the derived speaker embedding is language-agnostic.

Articulatory and Universal Phonetic Modeling

Using localized graphemes or language-bound phoneme dictionaries binds the text representation to specific accents. Advanced systems replace standard text inputs with language-independent phonetic frameworks:

• International Phonetic Alphabet (IPA) with Articulatory Vectors: Decomposing speech into universal articulatory features (e.g., place and manner of articulation, nasality, voicing) standardizes phonetic targets across disparate languages.
• Self-Supervised Speech Representations (SSL): Pre-trained cross-lingual models like multilingual HuBERT or WavLM produce robust acoustic representations that map similar sounds from different languages to shared latent spaces, minimizing regional pronunciation biases.

Decoupled Prosody and Duration Modeling

Accents are not merely localized to individual phoneme pronunciations; prosodic rhythms, lexical stress, and pitch contours strongly convey accent identity. When a system models pitch (\(F_0\)) and phoneme duration alongside speaker embeddings, the foreign speaker’s native prosody is often replicated in the target language. To fix this:

• Explicit prosody predictors are conditioned solely on the target language embedding and phoneme input, deliberately excluding the speaker embedding.
• Rhythm and duration are generated according to the target language's normative distributions rather than the reference speaker's source cadence.