Self-Supervised Learning for Low-Resource TTS

Building high-quality Text-to-Speech (TTS) systems traditionally requires tens of hours of studio-recorded, precisely transcribed audio. For thousands of low-resource languages, such datasets do not exist. Self-supervised learning (SSL) models trained on raw, untranscribed speech—such as wav2vec 2.0, HuBERT, and WavLM—capture rich phonetic, prosodic, and acoustic structures without human supervision. By repurposing these pre-trained representations, researchers can bypass the need for extensive paired text-audio corpora, effectively bootstrapping intelligible and natural speech synthesis for under-resourced languages with only a fraction of the traditional data requirements.

The Low-Resource TTS Challenge

Standard neural TTS architectures, such as FastSpeech 2 or Tacotron 2, rely on aligned pairs of text and high-fidelity speech to learn the complex mapping from characters or phonemes to acoustic frames (e.g., mel-spectrograms). In low-resource settings, this pipeline fails due to three main bottlenecks:

  1. Lack of Phonetic Aligners: Robust grapheme-to-phoneme (G2P) tools and forced aligners rarely exist for niche or indigenous languages.
  2. Data Scarcity: Available audio is often limited to a few minutes or hours rather than the dozens of hours required.
  3. Audio Quality: Sourced recordings often contain background noise, reverberation, and variable recording conditions.

Extraction of Discrete Acoustic Units

A primary method for leveraging SSL models in low-resource TTS involves discretizing continuous audio features into pseudo-phonetic units.

Intermediate layers of models like HuBERT or wav2vec 2.0 contain strong phonetic information while discarding background noise. Applying clustering algorithms (typically k-means) to these continuous frame-level representations creates a discrete sequence of "speech tokens" or "units."

Instead of predicting high-dimensional mel-spectrograms directly from text, the TTS acoustic model learns to predict these discrete units. This reformulates acoustic modeling into a sequence-to-sequence translation or autoregressive language modeling task:

Because the discrete units effectively act as an automatically discovered phoneme inventory, the system eliminates the dependency on expert-designed phonetic dictionaries or G2P systems.

Multilingual Cross-Lingual Transfer

SSL models trained on massive, multilingual raw audio datasets (such as XLSR or mSLAM) learn shared cross-lingual representations. These models map structurally similar acoustic properties across diverse languages into a shared latent space.

To bootstrap a target low-resource language:

  1. Shared Encoder Pre-training: The text and acoustic encoders are pre-trained on high-resource languages using multilingual SSL features.
  2. Representation Alignment: Because the SSL latent space captures universal phonetic structures, the model can align text from a low-resource language to SSL speech units using minimal paired supervision (often less than 30 minutes of paired data).
  3. Zero-Shot or Few-Shot Adaptation: Unseen phonemes or syllables from the low-resource language often borrow acoustic proximity from related high-resource phonemes within the shared latent space, resulting in stable pronunciation without catastrophic failure.

Unit-to-Speech Synthesis via Neural Vocoders

Once the acoustic model predicts discrete SSL units from text, these representations must be converted back into raw audio waveforms.

In low-resource scenarios, training a vocoder from scratch on target-speaker data produces severe artifacts. Instead, a unit-based neural vocoder (such as a modified HiFi-GAN) is pre-trained on multi-speaker, high-resource datasets to map discrete SSL units directly to waveforms.

Because the discrete SSL units abstract away speaker identity to a high degree, the vocoder can be conditioned on speaker embeddings (e.g., d-vectors) to synthesize the target speaker's voice, enabling high-fidelity audio generation without ever training an acoustic vocoder on the low-resource language.

Semi-Supervised Bootstrapping Loops

When small amounts of paired data are available alongside larger pools of untranscribed speech, iterative pseudo-labeling provides an additional performance boost:

  1. Train an initial low-resource TTS model using SSL-derived discrete units and the small paired set.
  2. Synthesize audio from raw text or use an SSL-based Automatic Speech Recognition (ASR) system to transcribe the raw audio.
  3. Filter synthesized or transcribed pairs based on confidence scores derived from the SSL representation space.
  4. Retrain the TTS system on the expanded dataset.

Key Advantages in Practice