Conditioning Vibrato and Ornamentation in Singing TTS

Singing-oriented Text-to-Speech (TTS), commonly referred to as Singing Voice Synthesis (SVS), generates vocal tracks by synthesizing linguistic content while adhering to musical scores. A primary challenge in SVS is reproducing expressive vocal nuances—specifically vibrato and stylistic ornamentation such as trills, runs, and pitch scoops—without distorting phonetic clarity. This article explores how modern deep learning architectures extract, parameterize, and condition vibrato and ornamentation features alongside standard phonetic inputs to produce natural, expressive singing.

Decomposition of Fundamental Frequency (\(F_0\))

Singing voice synthesis architectures typically decompose the fundamental frequency (\(F_0\)) contour into two primary components: a coarse macro-pitch and a fine micro-pitch. The macro-pitch represents the canonical musical score (note pitch and duration), while the micro-pitch captures expressive pitch deviations, including vibrato (periodic modulation of frequency and amplitude) and ornamentation (rapid transitions, portamentos, and embellishments).

To condition these elements, systems extract residual contours by subtracting the quantized score-level pitch from the ground-truth \(F_0\). From this residual:

• Vibrato parameters are explicitly derived using frequency-domain analysis (such as Short-Time Fourier Transform or Hilbert transforms) to estimate vibrato depth (extent in semitones) and rate (cycles per second, typically 4–7 Hz).
• Ornamentation parameters are captured as continuous temporal pitch deviations or discretized into stylistic tokens representing techniques like pitch bends, appoggiaturas, or mordents.

Multi-Stream Representation and Feature Alignment

Phonetic inputs and musical features exist at different temporal resolutions. Phonemes provide linguistic identity, usually mapped to phoneme-level embeddings, whereas vibrato and ornamentations operate continuously across acoustic frames or locally across specific note segments.

To reconcile these representations:

1. Duration Modeling: Phoneme durations are aligned with musical note boundaries using forced aligners or dedicated duration predictors.
2. Expansion to Frame Level: Phoneme embeddings and musical note embeddings (MIDI pitch and duration) are expanded to match the acoustic frame rate (e.g., through length regulators).
3. Temporal Mapping of Modulations: Vibrato and ornamentation representations are projected into the same temporal dimension. While a note's phoneme may remain static (e.g., an extended vowel), the associated pitch modulation contour fluctuates dynamically over time.

Conditioning Architectures in Acoustic Models

Modern acoustic models—such as FastSpeech-style transformers, continuous normalizing flows, and diffusion models (e.g., DiffSinger)—integrate expressive pitch features with phonetic vectors using several conditioning paradigms:

• Concatenative and Additive Fusion: The simplest method involves projecting phonetic embeddings, musical note embeddings, and frame-level pitch residual embeddings into a unified latent space. These vectors are added together with positional encodings or concatenated along the channel dimension before entering the acoustic decoder.
• Feature-wise Linear Modulation (FiLM): Rather than standard addition, expressive parameters can dynamically scale and shift the activation states of phonetic representations. Conditioning layers compute affine transformation parameters (\(\gamma, \beta\)) from the vibrato and ornamentation vectors, applying them to the intermediate representations of phonetic encoders: \[\text{FiLM}(h) = \gamma(c) \odot h + \beta(c)\] where \(h\) is the phonetic hidden state and \(c\) represents the expressive conditioning context.
• Cross-Attention Mechanisms: In dual-stream architectures, phonetic features and pitch modulation features pass through separate encoder pathways. Cross-attention layers enable the acoustic decoder to query phonetic identity while attending to expressive ornamentation contours, preventing the high-frequency temporal variations of vibrato from smearing formant transitions or altering vowel identity.

Generative Modeling of Residual Ornamentation

Relying solely on deterministic, explicit parameters can lead to robotic or static pitch modulations. Advanced SVS systems integrate probabilistic frameworks to condition expressive variation alongside phonemes:

• Variational Autoencoders (VAEs): Pitch-residual VAEs encode real singer ornamentations into a continuous latent space during training. At inference, the system samples style vectors from this latent distribution, conditioned on the phonetic sequence and musical context, yielding natural variation in vibrato onset, depth progression, and note transitions.
• Diffusion and Flow-Matching Models: These architectures model the complex distribution of micro-pitch and spectral details directly. The phonetic sequence, note pitch, and global singer identity serve as prior conditioning signals. The diffusion denoiser then iteratively reconstructs the fine-grained acoustic frames, naturally rendering coherent vibrato envelopes and ornamental flourishes that organically align with the sustained phonemes of the score.