Why Phoneme Mismatch Causes Accents in Cross-Lingual TTS

In cross-lingual Text-to-Speech (TTS), synthesis models often produce noticeable foreign accents when generating speech in a target language using a voice trained on a different source language. This article examines how phoneme inventory mismatch—the fundamental discrepancy between the distinct sound units of the source and target languages—forces models to approximate missing sounds and leak native acoustic traits, ultimately generating accented synthetic audio.

The Nature of Phoneme Inventory Mismatch

Every language relies on a unique inventory of phonemes, the smallest distinct units of sound used to build words. For instance, the English language contains phonemes like the dental fricative /θ/ (as in "think"), which does not exist in standard French, German, or Japanese.

When a TTS model is trained exclusively or predominantly on a speaker of Language A, the model learns the exact articulatory mechanics, formant frequencies, and vocal tract configurations of Language A's phoneme inventory. When prompted to synthesize speech in Language B, the system encounters target phonemes that the speaker's acoustic profile has never produced.

Phonemic Approximation and Substitution

When a TTS system faces a phoneme absent from the source speaker's inventory, it must resolve the missing information through approximation or direct mapping.

• Phoneme Mapping: Systems that rely on International Phonetic Alphabet (IPA) mappings often substitute an out-of-vocabulary phoneme with the acoustically closest native equivalent. For example, substituting the English /θ/ with /s/ or /t/.
• Shared Embedding Spaces: In end-to-end multilingual systems, unified phoneme embeddings attempt to place similar cross-lingual sounds into a shared latent space. However, if the speaker embedding has only ever been conditioned on the native phoneme distribution, the decoder projects the target phoneme using the acoustic dimensions of the source language.

This substitution or projection mimics the exact mechanism of human second-language acquisition: the speaker substitutes their native phonological categories for unfamiliar foreign sounds, resulting in a distinct non-native pronunciation.

Acoustic and Articulatory Bias

A voice embedding does not merely capture vocal pitch; it captures the speaker's habitual articulatory dynamics. These dynamics govern:

• Vowel Formants: The precise tongue height and mouth shape used to create vowel sounds differ subtly across languages, even for phonemes that share the same phonetic label.
• Coarticulation Effects: The way a sound transitions into the next sound depends on language-specific speech habits. A speaker's acoustic model implicitly encodes these native coarticulation patterns.
• Voice Onset Time (VOT): The timing between the release of a consonant and the start of vocal cord vibration varies widely between languages for the same plosives (e.g., /p/, /t/, /k/).

When synthesizing a target language, the model applies the source speaker's learned articulatory constraints to the new phoneme sequence. This structural bias causes vowels to sound off-target and consonants to carry non-native timing.

Prosodic Incompatibility

Phoneme inventory mismatches also disrupt suprasegmental features, including stress, duration, and pitch contours. Languages use different rules for phoneme duration and lexical stress (such as tone-timed versus stress-timed cadences).

Because the acoustic model relies on the source speaker's statistical priors, it frequently predicts phoneme durations and pitch trajectories aligned with the native language's rhythm rather than the target language's natural cadence. This mismatch in prosody compounds the phonetic errors, making the synthetic accent sound even more pronounced to native listeners.