Harmonic and Noise Separation in Neural Vocoders

Neural source-filter (NSF) vocoders bridge classical acoustic theory and modern deep learning by explicitly dividing speech synthesis into periodic (harmonic) and aperiodic (noise) components. In Text-to-Speech (TTS) pipelines, this explicit decomposition allows neural models to strictly adhere to fundamental frequency (\(F_0\)) contours while independently modeling breathiness, fricatives, and unvoiced consonants. By feeding mathematically defined excitation signals into dedicated neural filtering networks, NSF vocoders eliminate pitch-tracking errors, suppress metallic artifacts, and produce highly natural speech waveforms.

The Source-Filter Framework in Deep Learning

Classical speech production theory models the human voice as an excitation source modulated by the resonances of the vocal tract. Traditional neural vocoders often act as black boxes, learning to generate raw audio directly from mel-spectrograms without structural constraints. This frequently leads to pitch drift, buzzing, or phase collapse.

Neural source-filter models resolve this by embedding the physical separation of the source (vocal folds) and filter (vocal tract) directly into the neural architecture. Instead of generating the entire acoustic waveform from scratch, the neural network acts as a time-variant, nonlinear filter operating on predefined excitation signals.

Explicit Excitation Generation

The explicit separation begins at the excitation stage, where the model creates two distinct signals based on input conditioning features:

Harmonic Source (Voiced Component): The harmonic excitation represents the vibration of the vocal cords during voiced sounds like vowels. It is generated algorithmically from the fundamental frequency (\(F_0\)) track. The vocoder computes the instantaneous phase of the fundamental pitch and generates a deterministic signal, typically composed of a pulse train or a sum of sine waves matched to the fundamental frequency and its harmonics: \[e_{\text{harmonic}}(t) = \sum_{k=1}^{K} \sin\left(2\pi k \int_0^t F_0(\tau)d\tau + \phi_k\right)\] Because this signal is derived mathematically from \(F_0\), the model cannot deviate from the intended pitch.
Noise Source (Aperiodic Component): The unvoiced excitation represents air turbulence, friction, and aspiration. It is generated independently of pitch using stochastic distributions, such as zero-mean Gaussian white noise: \[e_{\text{noise}}(t) \sim \mathcal{N}(0, \sigma^2)\] This signal contains a flat, continuous spectrum ideal for producing unvoiced stops, whispered sounds, and the turbulent components of fricatives.

Dual-Stream Neural Filtering

Once generated, the harmonic and noise excitations are routed through specialized neural filter branches conditioned on mel-spectrograms or linguistic embeddings.

The Harmonic Neural Filter: This branch processes the deterministic sine-based signal. Dilated convolutional layers or recurrent blocks shape the excitation into vowel formants and voiced consonants, applying resonances and spectral envelopes that match the conditioning features.
The Noise Neural Filter: Operating in parallel, this branch takes the white noise excitation and applies spectral shaping to form aperiodic acoustic phenomena, such as the high-frequency energy in sounds like /s/, /f/, and /t/, as well as natural vocal breathiness.

Because each neural sub-network receives a dedicated, structurally pure input, neither network needs to learn how to create pitch or stochasticity from scratch. The filters strictly learn transfer functions.

Waveform Recombination

The outputs of the two filtered paths are combined to generate the final synthetic waveform. In most implementations, this occurs via direct linear addition:

\[y(t) = y_{\text{harmonic}}(t) + y_{\text{noise}}(t)\]

In advanced variants, dynamic gain networks predict time-varying weighting coefficients that balance the contribution of each branch at every sample or frame. During fully voiced segments, the harmonic branch dominates; during unvoiced speech or silence, the noise branch is either shaped into unvoiced consonants or attenuated to zero.

Practical Benefits in TTS Pipelines

Explicit harmonic-noise separation yields major advantages over unconstrained neural vocoders:

Pitch Accuracy: Voicing artifacts and pitch instability are virtually eliminated because the harmonic source strictly follows the input \(F_0\).
High-Frequency Naturalness: Separating noise allows high-frequency consonants to remain crisp and distinct, avoiding the muffled or phase-smeared quality common in purely autoregressive or generative adversarial architectures.
Efficient Generalization: NSF models generalize better to unseen speakers and extreme pitch shifts, requiring significantly less training data to achieve stable acoustic control.