Sub-Band Coding in Neural TTS Vocoders

Sub-band coding optimizes computational efficiency in neural Text-to-Speech (TTS) vocoders by decomposing high-rate, full-band audio signals into multiple narrow frequency bands with lower sampling rates. By predicting these lower-rate sub-bands in parallel and subsequently reconstructing the full-band waveform using digital filter banks, vocoders drastically reduce mathematical operations and latency without significantly degrading perceived audio quality. This article explores the mechanics of frequency spectrum division, downsampling, and parallel synthesis in modern neural vocoding architectures.

The Computational Bottleneck in Neural Vocoding

Standard high-fidelity speech typically requires a sampling rate of 24 kHz, 44.1 kHz, or 48 kHz. In a raw waveform generation model, synthesizing one second of audio involves predicting tens of thousands of sequential or spatially deep samples. Whether utilizing autoregressive networks, diffusion models, or generative adversarial networks (GANs), full-band waveform modeling demands heavy computational overhead, making real-time inference challenging on edge devices or resource-constrained servers.

Frequency Decomposition via Analysis Filter Banks

Sub-band coding mitigates this issue by dividing the total frequency spectrum into \(N\) distinct sub-bands before neural processing. This division is commonly achieved using an analysis filter bank, such as a Pseudo-Quadrature Mirror Filter (PQMF) bank.

The analysis filter bank splits the incoming acoustic target into discrete spectral regions:

1. Low-frequency bands: Capture fundamental frequencies (\(F_0\)) and primary formants essential for speech intelligibility.
2. High-frequency bands: Capture fricatives, sibilance, and ambient breathiness.

Because each sub-band covers only \(1/N\) of the total bandwidth, the Nyquist-Shannon sampling theorem allows each band to be downsampled (decimated) by a factor of \(N\). For example, an audio signal originally sampled at 24 kHz split into 4 equal sub-bands yields four distinct signals, each sampled at just 6 kHz.

How Spectrum Division Optimizes Neural Generation

Dividing and downsampling the spectrum yields multiple architectural and computational advantages during model inference:

• Temporal Reduction: By reducing the effective sequence length by a factor of \(N\), temporal convolutions or recurrent loops operate over far fewer steps per second of audio.
• Parallel Processing: Neural vocoders (such as Multi-band MelGAN or Sub-band WaveRNN) synthesize all \(N\) sub-bands simultaneously. Instead of generating a single 24 kHz waveform sequentially, the network outputs four 6 kHz streams concurrently, distributing the workload across parallel hardware threads or tensor cores.
• Reduced Receptive Field Requirements: At lower temporal resolutions, dilated convolutions cover larger time spans with fewer layers, allowing models to use shallower networks with fewer parameters while still capturing long-range acoustic dependencies.
• Efficient Memory Footprint: Processing shorter sequence representations significantly lowers activation memory usage during forward passes, enabling higher batch sizes and faster generation throughput.

Reconstruction via Synthesis Filter Banks

Once the neural network predicts the lower-rate sub-bands from input acoustic features (such as mel-spectrograms), the final step is signal reconstruction.

Rather than relying on a heavy neural network to upsample the audio back to the full resolution, the vocoder passes the predicted sub-bands through a non-trainable, mathematically deterministic synthesis filter bank (such as an inverse PQMF). The synthesis filter bank upsamples the sub-bands, applies phase-canceling reconstruction filters to neutralize spectral leakage and aliasing between adjacent bands, and sums them into a single, high-fidelity full-band waveform.

Because the synthesis filter bank consists of lightweight mathematical operations, this final step takes negligible compute time, achieving real-time or faster-than-real-time performance while retaining near-original audio fidelity.