Diphthong Trajectory in Parametric vs Neural TTS

Diphthong trajectory modeling represents a significant hurdle in modern speech synthesis because diphthongs involve dynamic, continuous formant shifts rather than static acoustic targets. This article examines the core difficulties of synthesizing these complex vowel glides, contrasting how parametric neural Text-to-Speech (TTS) architectures struggle with over-smoothing and intermediate feature collapse against how non-parametric, end-to-end neural models face obstacles with phase consistency, temporal alignment, and fine-grained latent path stability.

A diphthong is a single syllabic nucleus where the vocal tract transitions smoothly from an initial vowel posture to a final off-glide (such as /aɪ/ or /oʊ/). This movement produces non-linear, time-varying trajectories in the lower formant frequencies (primarily F1 and F2). Accurately synthesizing a diphthong requires precise control over transition onset, glide velocity, and offset timing, all of which vary significantly based on speaking rate, stress, and phonetic context.

In parametric neural TTS systems—where an acoustic model predicts intermediate representations like mel-spectrograms or vocoder parameters—the primary challenge is the over-smoothing effect. Acoustic models trained with regression objectives such as Mean Squared Error (MSE) or L1 loss inherently penalize outliers, favoring the statistical mean of the training data. Because human speakers produce variable glide rates, this statistical averaging blurs the sharp, non-linear formant trajectories into a muddy, static intermediate state. As a result, the synthetic diphthong often sounds like a single monophthongized vowel rather than a distinct, articulate glide. Furthermore, frame-based representations discretize time (typically in 10–12.5 ms hops), which can fail to capture rapid, micro-prosodic articulator movements during fast speech.

Non-parametric neural TTS systems—including direct waveform generators, flow-matching systems, and fully end-to-end models like VITS—bypass explicit parametric intermediate representations to model the speech distribution directly. While this prevents the spectral over-smoothing typical of parametric models, it introduces distinct challenges:

• Latent Trajectory Drift: Generative models relying on stochastic differential equations or diffusion processes must infer complex continuous paths in high-dimensional latent space. Without explicit phonetic trajectory constraints, the model can suffer from path drift, resulting in unnatural diphthong durations, unsteady pitch, or fluctuating vowel quality during the transition.
• Phase Incoherence and Jitter: Diphthong transitions demand rapid, continuous shifts in instantaneous frequency and pitch harmonics. Non-parametric models generating raw audio directly can produce subtle phase mismatches across the glide, leading to audible roughness, metallic artifacts, or buzzy transitions.
• Data Sparsity in Continuous Transitions: While a parametric system explicitly enforces acoustic features, non-parametric networks must learn every nuanced spectral shift entirely from the training distribution. Dialectal variations and rare diphthongs with limited training samples frequently result in inconsistent transition velocities or unnatural vowel off-glides.

Ultimately, the difficulty in parametric neural TTS stems from representational bottlenecks and objective functions that wash out rapid spectral dynamics. Conversely, non-parametric neural TTS struggles with the unconstrained nature of end-to-end generation, where preserving phase coherence, fine-grained duration control, and temporal alignment across non-linear spectral glides demands immense data scale and tightly regulated latent sampling.