Causal Convolution in Streaming TTS Vocoders

Streaming text-to-speech (TTS) systems require real-time audio synthesis with minimal latency, prompting the widespread adoption of causal convolutions within neural vocoders. This article explores how replacing standard, bidirectional convolutions with causal variants alters the network's receptive field, enforces strict temporal dependencies, and impacts synthesized acoustic quality. It also covers common architectural compromises and mitigation techniques used to maintain high-fidelity speech while preserving streaming feasibility.

Understanding Causal Convolutions

In standard convolutional neural networks for audio synthesis, such as offline HiFi-GAN or WaveNet, convolutions are non-causal. The output at time step \(t\) is computed using both past frames (\(t - k\)) and future frames (\(t + k\)). This bidirectional context provides comprehensive information about upcoming phonemes, transitions, and phase trajectories.

Causal convolution strictly enforces the flow of time by masking or padding input sequences so that the calculation at time \(t\) depends exclusively on inputs at time steps \(\le t\). In a streaming vocoder, this design prevents the model from needing to wait for future audio frames, enabling chunk-by-chunk or sample-by-sample audio generation with minimal buffering.

Impact on Receptive Fields

The receptive field of a neural vocoder defines the window of input frames that influence the generation of a specific audio sample. Enforcing causality introduces distinct constraints on how this field is structured:

• Asymmetric Expansion: Unlike non-causal networks where the receptive field expands symmetrically in both directions, a causal receptive field expands exclusively backward into the past.
• Reduced Effective Context: Because future context is completely eliminated, a causal filter of kernel size \(K\) has half the immediate contextual scope per layer compared to a centered filter of the same size.
• Dependence on Dilations: To compensate for the lack of forward context, streaming vocoders must rely heavily on deep stacks of dilated causal convolutions. Dilations expand the historical receptive field exponentially without increasing parameter count, ensuring the model captures long-term pitch periods and harmonic structures.

Effects on Acoustic Quality

Restricting the receptive field to past context directly degrades acoustic fidelity if not properly addressed:

• Phonetic and Prosodic Smearing: Speech is characterized by co-articulation, where current vocal tract positions are heavily influenced by upcoming sounds. Without future context, vocoders struggle to predict clean formant transitions, resulting in muffled consonants, muffled transients, or slight unnaturalness at phoneme boundaries.
• Phase Inconsistencies: High-fidelity waveform generation depends on coherent phase reconstruction. Future context assists non-causal vocoders in smoothing pitch contours and avoiding phase discontinuities. Causal models frequently exhibit phase jitter, metallic artifacts, or buzzy degradation, particularly in unvoiced segments.
• Error Propagation: Because causal systems generate outputs incrementally based on previous hidden states and inputs, any intermediate distortion can propagate forward across frames, causing audible drift or instability in pitch.

Mitigating Quality Loss in Streaming Architecture

Engineers and researchers employ several architectural strategies to narrow the quality gap between causal and non-causal vocoders:

• Bounded Lookahead (Semi-Causal Layers): Instead of pure causality, architectures often introduce a small, fixed lookahead window (typically 5 to 20 milliseconds). This small delay remains imperceptible to human users in real-time interactions but provides the receptive field with sufficient forward context to smooth phonetic transitions.
• Larger Receptive Depths: Increasing the number of residual blocks or dilation cycles backward ensures that historical context encompasses multiple fundamental pitch cycles, stabilizing harmonic generation.
• Conditioning Enhancements: Providing stronger auxiliary inputs, such as explicit fundamental frequency (\(F_0\)) values, energy contours, or phoneme alignments extracted from the acoustic model, helps the causal vocoder resolve ambiguities without needing to infer future states directly from the audio history.