Tone Sandhi Challenges in Multilingual TTS

Tone sandhi—the phonological phenomenon where a syllable's pitch contour changes based on the tones of neighboring syllables—presents a major hurdle for acoustic modeling in multilingual Text-to-Speech (TTS) systems. In East Asian languages such as Mandarin, Min Nan, Wu, and Vietnamese, these contextual tonal shifts break the one-to-one mapping between base lexical tones and actual surface acoustic realizations. In a shared multilingual architecture, this dynamic creates front-end prediction errors, Fundamental Frequency (\(F_0\)) modeling conflicts, data sparsity, and cross-lingual negative transfer between tonal and non-tonal languages.

Phonological Divergence and Front-End Errors

Acoustic models rely heavily on the phonemic input generated by the text-processing front end (Grapheme-to-Phoneme, or G2P). In languages with complex tone sandhi, the written characters reflect canonical (lexical) tones rather than spoken (surface) tones. For instance, in Mandarin, two consecutive third tones cause the first to shift into a second tone, while in Southern Min (Taiwanese Hokkien), almost every syllable in a non-final syntactic position undergoes sandhi according to complex cyclic rules.

If the front-end fails to resolve these rules prior to feeding phoneme sequences to the acoustic model, the model receives contradictory target labels. An acoustic model attempting to synthesize a word will receive a label for Tone A while observing the acoustic profile of Tone B in the training audio, resulting in blurred, averaged, or unnatural pitch targets.

Syntactic and Prosodic Dependencies

Tone sandhi is rarely purely phonological; it is heavily dependent on syntax, semantics, and speech tempo:

• Prosodic Hierarchy: Sandhi domains are constrained by prosodic word and phrase boundaries. Two syllables that might trigger sandhi in isolation will not do so if separated by a major prosodic break.
• Speech Rate: At faster speech tempos, sandhi rules often cascade across broader syntactic boundaries, creating surface contours that do not appear in slower, canonical reference speech.

Multilingual acoustic architectures typically generalize prosodic modeling across languages. Because non-tonal languages (such as English or Spanish) use pitch primarily for sentence-level intonation rather than lexical meaning, unified prosodic decoders often fail to apply the fine-grained structural constraints required to delineate language-specific sandhi boundaries.

Continuous \(F_0\) Contour Modeling

Acoustic models must predict continuous \(F_0\) contours alongside spectral features. Tone sandhi causes dramatic shifts in pitch height and trajectory over very short temporal windows. In multilingual TTS:

• Shared Encoders: When a single acoustic model processes both tonal and non-tonal languages, the shared representation space risks "tone flattening." The model may favor smooth, global intonational drift over the localized, sharp \(F_0\) excursions intrinsic to tonal sandhi shifts.
• Coarticulation Conflicts: In tonal languages, the realized pitch is a combination of underlying tone, sandhi transformation, and physiological coarticulation. Disentangling sandhi-induced pitch shifts from standard phonetic transitions requires high-capacity conditioning that multilingual models often fail to isolate per language.

Negative Transfer Across Languages

Multilingual TTS models share sub-networks (such as encoders, attention mechanisms, or diffusion decoders) to enable zero-shot voice cloning and reduce overall parameter footprint. However, tone sandhi introduces negative transfer:

1. Embedding Contamination: Language or speaker embeddings may inadvertently absorb tone variations. Instead of the model recognizing that a pitch drop is an obligatory sandhi rule, it may misattribute the shift to speaker idiosyncrasy or emotional coloring.
2. Gradient Dilution: During training, gradients derived from non-tonal data penalize steep, local \(F_0\) changes to preserve smoothness. This directly penalizes the acoustic model's ability to render abrupt sandhi transitions accurately in languages like Hakka or Cantonese.

Data Sparsity in Sandhi Permutations

While base tones may have balanced distributions in a training corpus, the permutations of multi-syllable sandhi clusters follow a severe long-tail distribution. Complex Sandhi systems (such as those in Min and Wu dialects) feature extensive multi-tone interaction matrices.

In low-resource East Asian languages, many valid sandhi combinations never appear in the training corpus. When a multilingual model encounters an unseen combination at inference, it cannot reliably interpolate the acoustic contour, often defaulting to either an incorrect lexical tone or an unnatural pitch plateau. Robust modeling requires either explicit sandhi-aware phonological conditioning or auxiliary pitch loss functions tailored specifically to East Asian tonal geometry.