How Multimodal LLMs Unify Text and TTS Synthesis

Modern multimodal Large Language Models (LLMs) bridge the gap between text generation and speech synthesis by converting continuous acoustic signals into discrete tokens, allowing a single transformer backbone to treat speech generation identically to language modeling. By integrating neural audio codecs with unified vocabulary schemes, these models eliminate the need for disjoint, multi-stage Text-to-Speech (TTS) pipelines. Instead, they process, reason over, and synthesize both text and audio end-to-end within a single autoregressive sequence-to-sequence framework.

Discrete Audio Tokenization

The foundation of unifying text and audio synthesis is discrete representation. Raw audio waveforms are continuous and high-frequency, making them unsuitable for standard transformer architectures. Modern architectures rely on Neural Audio Codecs—such as EnCodec, SoundStream, or Descript Audio Codec (DAC)—to compress audio into discrete representations.

These codecs use an encoder-decoder architecture with Residual Vector Quantization (RVQ). The encoder compresses the continuous waveform into a lower frame-rate latent space, and the RVQ quantizes these vectors across multiple hierarchical codebooks to preserve fine acoustic details. The resulting discrete codebook indices serve as "audio tokens," structurally equivalent to subword text tokens.

Unified Vocabulary and Shared Embedding Space

To process both modalities within a single backbone, the model's embedding layer is expanded to include both textual tokens and acoustic tokens:

Because both text and audio indices map to vectors of the same hidden dimension, the transformer's self-attention mechanisms operate agnostic to the underlying modality.

Sequence Modeling and Codebook Flattening

Text generation typically involves a single sequence of tokens, whereas high-fidelity neural codecs use multiple parallel RVQ codebooks per audio frame. Unified architectures reconcile this structural difference using specific sequence modeling strategies:

  1. Flattened Interleaved Sequences: The model flattens the multi-codebook structure into a single sequence by interleaving tokens, processing the first codebook level (coarse semantic details) followed by subsequent levels (fine acoustic details) for each time step.
  2. Delay Patterns: The model introduces an intentional temporal offset across codebook levels. This allows the transformer to predict dominant acoustic tokens first while conditioning finer acoustic layers on slightly earlier timeframes.
  3. Dual-Track or Hierarchical Generation: The main autoregressive backbone generates text and primary semantic/acoustic tokens, while a smaller integrated feed-forward or non-autoregressive sub-transformer predicts the remaining parallel RVQ levels in a single forward pass.

End-to-End Generation Pipeline

During text-to-speech inference, the model operates through a continuous autoregressive loop:

  1. Prompt Processing: The input text prompt (and optional voice-conditioning audio prefix) is tokenized and embedded.
  2. Joint Reasoning: The transformer generates an internal text response or reads the target text directly, attending across the context using standard causal self-attention.
  3. Modality Switching: Upon reaching a generation trigger token, the model shifts its prediction head to sample from the audio token distribution.
  4. Acoustic Token Generation: The transformer autoregressively predicts the sequence of codec tokens, maintaining prosody, speaker characteristics, and linguistic content derived from the textual context.
  5. Waveform Reconstruction: The generated discrete audio tokens are extracted and passed directly through the neural codec's convolutional decoder (vocoder), which reconstructs the final high-fidelity audio waveform.

By standardizing audio as discrete tokens and handling multi-codebook alignment within the attention mechanism, a single transformer backbone simultaneously performs textual reasoning and acoustic synthesis.