Rhythm Alignment in Singing Synthesis vs TTS Duration

This article examines how Lyric-to-Singing (LTS) synthesis systems handle temporal alignment and rhythm compared to traditional duration predictors in Text-to-Speech (TTS). While standard TTS predicts flexible, context-driven durations to emulate natural speech cadence, singing synthesis must reconcile rigid musical constraints—such as meter, tempo, and note boundaries—with phonetic realities. The following breakdown highlights the architectural, phonetic, and conditioning distinctions that define rhythm management in both domains.

External Score Constraints vs. Statistical Cadence

Standard TTS duration models (such as those in FastSpeech or VITS) predict phone durations based purely on linguistic context, syntactic phrasing, and prosodic intent. The rhythm is unconstrained and continuous; there is no rigid time grid.

In contrast, lyric-to-singing systems must align phonemes to an explicit external musical score containing note pitches, onsets, offsets, and a global tempo (BPM). Instead of generating a rhythm from scratch, an LTS model must map phonetic material into predefined musical timeframes while maintaining synchronization with a musical beat.

Vowel Extension and Consonant Placement

Human speech and musical singing distribute time across phonemes differently:

• TTS Pacing: When a speaker slows down or speeds up, both consonants and vowels adjust in duration relatively proportionally, guided by typical linguistic stress patterns.
• Singing Pacing: Singing holds pitch primarily on vowels and voiced sonorants. When a syllable is extended across a long musical note, the vowel absorbs nearly all the temporal elongation. Consonants cannot simply be stretched, as doing so introduces unnatural slurring and degrades intelligibility.

LTS duration models resolve this by decoupling the syllable into onset consonants, the vowel nucleus, and coda consonants. The model generally assigns standard, speech-like durations to the consonants while dynamically allocating the remaining note duration to the vowel. Furthermore, LTS models must decide whether consonants are placed before the beat (pre-beat alignment) so the vowel starts precisely on the downbeat, or directly on the beat.

Handling Melisma and Multi-Note Phrasing

Standard TTS operates on a monotonic progression where each phoneme corresponds to a single continuous slice of time. Speech rarely requires a single phoneme to transition across distinct target pitches without a phonetic boundary.

Singing voice synthesis regularly encounters melisma, where a single lyric syllable spans across multiple musical notes and pitch transitions. LTS rhythm alignment architectures must split a single phoneme duration across multiple musical note events. This requires hierarchical duration modeling: first mapping lyric syllables across musical note boundaries, and then calculating sub-phonemic durations that preserve the required pitch transitions without introducing unintended breaks or artifacts.

Architecture and Conditioning Discrepancies

The input representations and alignment mechanics in LTS diverge sharply from TTS:

• Conditioning Vectors: TTS duration predictors typically accept phoneme embeddings, positional encodings, and speaker embeddings. LTS duration models require musical score features, including note duration, beat position within the measure, tempo, and slur flags.
• Alignment Mechanisms: TTS models increasingly use unsupervised alignment methods, like Monotonic Alignment Search (MAS), to learn duration implicitly from acoustic data. LTS systems frequently rely on forced alignment conditioned on explicit score timing or hierarchical duration predictors (note-level to phone-level) to ensure the generated voice does not drift away from the backing tempo over extended passages.