Why Autoregressive TTS on CPUs Fails at Scale

Running autoregressive Text-to-Speech (TTS) models entirely on CPU hardware becomes commercially and computationally unfeasible when scaling to production workloads. While central processing units (CPUs) handle general-purpose computing efficiently, autoregressive speech synthesis demands massive sequential matrix operations and rapid memory access that drastically exceed standard CPU architecture capabilities. As concurrency demands rise, CPU-only infrastructure suffers from crippling latency, catastrophic throughput degradation, and unsustainable operational costs.

The Sequential Generation Bottleneck

Autoregressive TTS models—such as Bark, Tortoise-TTS, and VALL-E—synthesize speech sequentially, predicting audio tokens one step at a time where each new token depends strictly on all previously generated tokens. Because token \(N\) requires the output of token \(N-1\), the generation process across the temporal dimension cannot be parallelized.

A typical speech segment requires hundreds or thousands of forward passes through the neural network to produce just a few seconds of audio. While graphics processing units (GPUs) accelerate individual forward passes using thousands of dedicated tensor cores, CPUs rely on a limited number of high-performance cores optimized for complex branching rather than brute-force matrix multiplication. Consequently, individual token generation on a CPU is inherently slow, leading to high initial latency and poor overall responsiveness.

Memory Bandwidth Limitations

Autoregressive inference is heavily memory-bandwidth bound. During each sequential step:

  1. The model must fetch its entire set of weights from RAM into the processor's cache.
  2. It computes the next token.
  3. It repeats this retrieval process for every single subsequent token.

Standard CPU platforms rely on DDR4 or DDR5 memory channels, providing throughput typically ranging between 50 GB/s and 300 GB/s across multiple channels. In contrast, modern accelerators equipped with High Bandwidth Memory (HBM) deliver anywhere from 1,000 GB/s to over 3,000 GB/s. Because CPUs lack the memory bus width necessary to shuffle multi-gigabyte weight tensors thousands of times per second, the processor cores spend substantial compute cycles idle, waiting for data retrieval.

Unfavorable Real-Time Factor (RTF)

Production voice interfaces demand a low Real-Time Factor (RTF)—the ratio of computation time to the duration of the audio produced. An RTF under 1.0 is required for real-time output, while streaming voice applications typically require an RTF below 0.2 to account for network jitter and buffering.

On CPU-only hardware, complex autoregressive TTS models frequently exhibit RTFs ranging from 2.0 to 10.0 or higher. Generating a 10-second voice clip can require 20 to 100 seconds of compute time. This computational drag prevents real-time, conversational interaction and leads to immediate queue buildup under standard user loads.

Concurrency and Infrastructure Costs

When scaling to serve multiple concurrent users, CPUs face steep performance degradation: