Designing Acoustic Codebooks for Emotional TTS

Modern neural Text-to-Speech (TTS) models increasingly rely on discrete acoustic codebooks to compress high-fidelity audio into manageable tokens. However, standard lossy vector quantization frequently discards subtle acoustic details—such as micro-prosody, vocal fry, and breathiness—that carry vital emotional nuance. Preserving these expressive characteristics requires a deliberate redesign of the quantization pipeline, focusing on disentangled representations, multi-scale residual quantization, and emotion-aware objective functions.

Disentangling Prosody from Phonetic Content

Standard neural audio codecs compress speech into a shared latent space where dominant linguistic patterns often suppress subtler emotional signals. To preserve emotional nuance, codebooks should be split into dedicated, disentangled streams:

• Prosodic Codebooks: Quantize low-frequency variations, including fundamental frequency (\(F_0\)) contours, energy trajectories, and duration patterns across syllable and sentence boundaries.
• Acoustic/Timbral Codebooks: Quantize spectral envelope details, vocal tract resonances, and paralinguistic cues such as trembling or breathy phonation.
• Linguistic Codebooks: Quantize canonical phonemic content independently, preventing phonetic reconstruction from overpowering subtle expressive variations.

Explicitly separating these features forces the quantization bottleneck to retain distinct representations of emotional state rather than discarding them as negligible reconstruction noise.

Structuring Hierarchical and Residual Quantization

Residual Vector Quantization (RVQ) is widely used to scale codebook capacity, but naive implementations allocate most bits to coarse timbral matching. Codebooks can be structured to preserve emotional hierarchy:

• Depth-Wise Allocation: Assign the initial quantization stages strictly to macro-prosody and emotional tone. Because emotional arousal and valence are primarily driven by macro-level intonation, anchoring the first codebook layers to these attributes ensures that even low-bitrate generation maintains the target emotion.
• Micro-Dynamic Capture in Later Stages: Use the deeper RVQ layers specifically for high-frequency paralinguistic artifacts (such as sighs, gasps, and vocal creak). Preventing aggressive codebook dropout during training ensures these emotional cues are not pruned.
• Temporal Multi-Scale Processing: Implement different frame rates across codebooks. A slower frame rate (e.g., 20–50 Hz) captures sentence-level emotional prosody, while a faster frame rate captures transient expressive nuances without ballooning the overall token count.

Implementing Emotion-Aware Optimization

Codebook learning depends on the loss landscape used during training. Standard reconstruction metrics—such as \(L_1\) mel-spectrogram distance and broad adversarial losses—favor intelligibility and perceptual sharpness over emotional fidelity. To safeguard expressive nuances, the training pipeline must incorporate dedicated emotional objectives:

• Perceptual Emotion Distance: Utilize pre-trained Speech Emotion Recognition (SER) models to compute perceptual feature-matching losses in the latent space of the codec. This penalizes the quantizer when it alters the emotional classification of the reconstructed output.
• Pitch and Energy Dynamics Loss: Apply explicit regression losses to extracted \(F_0\) and root-mean-square (RMS) energy contours, particularly penalizing deviations at inflection points where emotional transitions occur.
• Contrastive Expressive Loss: Use self-supervised contrastive learning to push speech segments with different emotional intensities into distinct regions of the codebook space, preventing expressive compression collapse.

Dynamic Bitrate and Codebook Capacity

Emotional speech features high variance in information density; a calm sentence requires fewer expressive bits than an impassioned exclamation. Employing variable-rate quantization or dynamic codebook routing allows the codec to allocate higher bit depths to emotionally volatile segments while compressing neutral passages more aggressively. This preserves critical emotional inflection without unnecessarily increasing the overall sequence length for downstream TTS generation.