Audio Text Forced Alignment Using HMM and CTC

Forced alignment is a critical preprocessing technique in Text-to-Speech (TTS) workflows that automatically pairs spoken audio recordings with their written transcriptions down to the word or phoneme level. By establishing precise time boundaries for speech segments, alignment algorithms provide the supervised temporal data required to train neural acoustic models and vocoders. This article examines the mechanics of forced alignment, comparing traditional statistical methods using Hidden Markov Models (HMMs) with modern deep-learning approaches powered by Connectionist Temporal Classification (CTC) loss.

The Purpose of Forced Alignment in TTS

Modern TTS architectures, particularly non-autoregressive models like FastSpeech 2, require frame-level or phoneme-level duration information to learn how long each sound should be held during synthesis. Manually annotating thousands of hours of audio is impractical. Forced alignment automates this by taking a verified text transcript, converting it to phonemes via a pronunciation dictionary or grapheme-to-phoneme (G2P) model, and mapping those phonemic states to specific time intervals in the corresponding audio.

Forced Alignment Using Hidden Markov Models (HMMs)

Hidden Markov Models have historically formed the backbone of speech recognition and alignment toolkits like Kaldi and the Montreal Forced Aligner (MFA).

Feature Extraction: Raw audio is sliced into short, overlapping windows (typically 20–25 milliseconds) to generate spectral features, such as Mel-Frequency Cepstral Coefficients (MFCCs) or filter bank energies.
Acoustic Modeling: Each phoneme is modeled as a sequence of hidden states (typically three states representing the beginning, middle, and end of the sound). In classical setups, Gaussian Mixture Models (GMMs) or Deep Neural Networks (DNNs) output the probability that a specific audio frame corresponds to an HMM state.
Constrained Search Space: Because the text transcript is already known, the system does not need to recognize unconstrained speech. Instead, the HMM transitions are strictly restricted to the linear sequence of phonemes defined by the transcript.
Viterbi Decoding: The Viterbi algorithm computes the most likely hidden state sequence through dynamic programming. It traverses the audio frames from start to finish, finding the single path that maximizes the joint acoustic and transition likelihood. The state transition points define the exact start and end timestamps for each phoneme.

Forced Alignment Using CTC Loss

Connectionist Temporal Classification eliminates the need for predefined state transitions by allowing neural networks to output sequence labels directly over variable-length inputs without frame-by-frame alignment annotations.

Frame-Level Predictions: A neural network (such as a Conformer, BLSTM, or pretrained model like wav2vec 2.0) processes acoustic features and generates a probability distribution over the vocabulary for every frame. The vocabulary includes standard characters or phonemes plus a special "blank" token (\(\epsilon\)), which indicates a lack of identifiable label emission or transitions between repeated tokens.
CTC Alignment Graph: A transcript like "CAT" can be represented by multiple valid frame sequences (e.g., C-C-A-A-T, C-ε-A-T-T, or ε-C-A-ε-T). CTC loss computes the sum of probabilities across all valid alignment paths that collapse into the target transcript when blanks and duplicate adjacent labels are removed.
Finding Timestamps: During forced alignment, dynamic programming (similar to the forward-backward algorithm) is used to locate the path with the highest probability constrained to the ground-truth sequence. The moments where the model shifts probability mass from one non-blank character to another denote the precise temporal boundaries between sounds.

HMM vs. CTC for TTS Preparation

Data Requirements: HMM-based aligners rely heavily on well-defined pronunciation lexicons and acoustic models, performing exceptionally well even on smaller, domain-specific corpora. CTC-based aligners rely on deep neural networks that typically require substantial pretraining data or existing foundational models to achieve peak accuracy.
Granularity and Robustness: HMMs explicitly enforce left-to-right state transitions, which prevents them from skipping segments of audio and makes them reliable for phoneme-level boundary precision. CTC networks tend to emit target tokens in compact bursts surrounded by blank tokens, which requires post-processing or specialized dynamic time warping to convert discrete emission spikes into continuous duration spans.
Speed and Modern Workflows: CTC-based alignment can run natively on GPUs within popular deep learning frameworks, allowing for rapid batch processing of large datasets. While HMM toolkits remain standard due to their battle-tested phoneme accuracy, CTC alignment provides a streamlined, fully differentiable alternative directly integrated into end-to-end Python pipelines.