Text-to-Speech as Audio Language Modeling

This article explores how transformer-based autoregressive models like Bark and AudioLM revolutionize text-to-speech (TTS) synthesis by framing speech generation as an audio language modeling problem. Rather than relying on traditional two-stage pipelines that generate intermediate representations like mel-spectrograms, these systems discretize continuous audio waveforms into discrete tokens using neural audio codecs. By organizing these tokens into hierarchical sequences, decoder-only transformers can generate expressive, natural speech, complete with realistic prosody, ambient sounds, and non-verbal vocalizations using standard next-token prediction techniques.

The Paradigm Shift: From Acoustic Features to Audio Tokens

Traditional neural TTS architectures typically decompose speech synthesis into two distinct stages: converting text to intermediate acoustic features (such as mel-spectrograms) via models like Tacotron or FastSpeech, followed by a separate neural vocoder (such as HiFi-GAN) to synthesize raw audio waveforms. While effective, this pipeline often struggles with zero-shot voice cloning, complex acoustic environments, and spontaneous, non-verbal vocal nuances.

Audio language modeling bypasses intermediate continuous representations entirely. It reconceptualizes audio as a sequence of discrete symbols, analogous to words or subword units in natural language processing (NLP). By compressing continuous audio into discrete representations using deep neural codecs, systems like Bark and AudioLM treat speech generation identically to text generation: predicting the next token in a sequence conditioned on preceding context.

Discretizing Audio with Neural Codecs

To treat audio like text, continuous 1D audio signals must be transformed into a compact, discrete token space. This is achieved using neural audio codecs, predominantly based on Residual Vector Quantization (RVQ), such as SoundStream or EnCodec.

• Residual Vector Quantization (RVQ): A single vector quantizer cannot compress high-fidelity audio into a small codebook without significant quality loss. RVQ resolves this by applying multiple stages of vector quantization. The first quantizer encodes the input feature vector, and each subsequent quantizer encodes the residual error from the previous stage. This yields a multi-stream sequence of discrete tokens per audio frame.
• Semantic vs. Acoustic Tokens: Systems like AudioLM distinguish between two types of audio information:
  • Semantic Tokens: Derived from self-supervised speech representations (such as w2v-BERT or HuBERT), these tokens strip out background noise, channel acoustics, and speaker identity, capturing pure linguistic meaning, phonetic context, and high-level prosody.
  • Acoustic Tokens: Extracted directly from RVQ-based neural codecs (such as SoundStream or EnCodec), these tokens retain speaker timbre, recording acoustics, room impulse response, and fine temporal details necessary for high-fidelity reconstruction.

The Hierarchical Architecture of AudioLM and Bark

Generating high-fidelity audio autoregressively presents a dimensional challenge: processing hundreds or thousands of tokens per second rapidly exhausts the context window of standard transformers. To mitigate this, models employ a hierarchical, multi-stage autoregressive framework.

The AudioLM Approach

AudioLM structures the generation process into three sequential stages, factorizing the joint probability distribution of the speech tokens:

1. Text/Audio to Semantic: The model maps the input prompt to a sequence of semantic tokens, capturing the broad linguistic content and structure.
2. Semantic to Coarse Acoustic: A transformer conditions on the generated semantic tokens to predict the first few levels of the RVQ acoustic codebook. These coarse acoustic tokens capture speaker identity, fundamental frequency (\(F_0\)), and general tonal characteristics.
3. Coarse to Fine Acoustic: A final non-autoregressive or masked language model conditions on the coarse acoustic tokens to predict the remaining fine RVQ levels. Fine tokens represent high-frequency details that do not require long-term context, making fully autoregressive generation unnecessary at this stage.
4. Decoding: The final token streams are passed directly into the neural codec's decoder to reconstruct the raw continuous waveform.

The Bark Approach

Suno's Bark utilizes a conceptually similar multi-tiered transformer structure that operates over EnCodec token streams:

1. Text to Semantic: A GPT-style transformer converts input text tokens directly into high-level semantic tokens (often quantized from HuBERT-like feature spaces).
2. Semantic to Coarse Acoustic: A second transformer generates the first two codebooks of EnCodec tokens based on the semantic sequence. This step determines voice characteristics, emotion, timing, and non-verbal vocalizations (such as laughing, sighing, or gasping).
3. Coarse to Fine Acoustic: A third transformer (typically an autoregressive or sequence-to-sequence model) expands the two coarse codebooks into the full eight codebooks required by EnCodec.
4. Waveform Synthesis: The full set of quantized acoustic vectors is fed into the EnCodec decoder to output the finished waveform.

Autoregressive Generation and In-Context Learning

By adopting the standard decoder-only transformer architecture (identical to models like GPT-4), audio language models inherit fundamental properties of large language models:

• In-Context Learning: Speaker adaptation, emotion, and acoustic environment can be conditioned without fine-tuning. By prepending a short audio prompt (represented as semantic and acoustic tokens) to the input sequence, the model natively continues the pattern, preserving the prompt's timbre, speaking rate, and background acoustics.
• Unified Multimodal Conditioning: Because text, semantic audio, and acoustic audio are all expressed as discrete tokens within a shared or interconnected vocabulary, conditioning on text, voice prompts, or style descriptors requires no specialized architectures—only token concatenation and attention masking.
• Emergent Non-Verbal Capabilities: Because training sets comprise unconstrained raw audio rather than isolated studio voice recordings, autoregressive models learn to sample non-speech human sounds natively, such as hesitation pauses ("um," "uh"), breathing patterns, laughter, and contextual throat clearing.

Key Trade-offs

While audio language modeling delivers unprecedented naturalness and zero-shot voice cloning capabilities, it introduces specific engineering trade-offs compared to non-autoregressive TTS systems:

• Inference Latency: Sequential token generation requires running the transformer forward pass for every single token step, making real-time streaming more computationally expensive than non-autoregressive diffusion or feed-forward models.
• Hallucinations and Stability: Much like text LLMs, autoregressive audio models can occasionally skip words, invent non-existent phonemes, loop repetitive audio artifacts, or abruptly drift in speaker timbre over long sequence generations.