Disaggregating Prefill and Decode in AI Inference Clusters

Large-scale artificial intelligence inference clusters increasingly separate the prefill and decode phases of large language model processing onto distinct hardware pools to maximize efficiency, reduce latency, and lower operational costs. While the prefill phase processes the initial user prompt in parallel and requires massive computational power, the decode phase generates responses one token at a time and relies heavily on memory bandwidth. Running both phases on the same servers creates resource bottlenecks and degrades user experience. By disaggregating these workloads onto specialized hardware tailored to their specific demands, infrastructure operators can eliminate resource contention, meet strict latency targets, and optimize hardware utilization across the entire cluster.

The Two Phases of LLM Inference

Large language model (LLM) inference consists of two distinct operational stages: the prefill phase and the decode phase. Each phase interacts with GPU hardware in fundamentally different ways.

The prefill phase takes the user's input prompt and processes all the tokens simultaneously. Because the system knows the entire input in advance, it can perform massive matrix-matrix multiplications in parallel. This makes the prefill phase compute-bound. The primary limitation is the raw processing speed (FLOPS) of the GPU, while memory bandwidth remains relatively underutilized.

The decode phase generates the output text sequentially, one token at a time. Each new token depends on the tokens generated before it, as well as the stored context from the prefill phase (known as the KV cache). Because the GPU only processes a single token per step for a given request, it performs matrix-vector operations rather than large matrix-matrix operations. For every single token generated, the GPU must load the entire model’s weights and the KV cache from High Bandwidth Memory (HBM) into the compute cores. This makes the decode phase memory-bandwidth-bound. The compute cores spend most of their time idle, waiting for data to arrive from memory.

The Problem with Colocated Inference

Traditionally, AI clusters have used a colocated architecture, where a single GPU instance handles both the prefill and decode phases for a request. This approach introduces severe inefficiencies at scale:

  • Resource Contention and the Convoy Effect: When a long prompt arrives for prefill, it monopolizes the GPU's compute cores. If that same GPU is currently decoding tokens for other users, those decode requests are paused or slowed down. This creates a "stutter" in text generation for active users.
  • Conflicting Batching Strategies: Prefill benefits from small batch sizes to keep the Time to First Token (TTFT) low. Conversely, decode requires large batch sizes to amortize the cost of loading model weights from memory, which maximizes throughput and keeps the Time Per Output Token (TPOT) low. Colocating them forces a compromise that degrades both metrics.
  • Over-provisioning Hardware: To meet strict latency Service Level Agreements (SLAs) for both TTFT and TPOT on shared hardware, operators must over-provision expensive GPUs, resulting in wasted compute capacity during decode and wasted memory bandwidth during prefill.

How Disaggregated Architecture Works

Disaggregated inference solves these problems by splitting the cluster into two dedicated pools: a Prefill Pool and a Decode Pool.

  1. Request Routing: When a user prompt enters the system, the load balancer sends it directly to the Prefill Pool.
  2. Prefill Execution: The Prefill Pool processes the prompt at maximum compute capacity, generating the very first output token and building the initial KV cache.
  3. KV Cache Transfer: Once the prefill is complete, the generated KV cache is transferred over a high-speed interconnect (such as NVLink or RDMA over InfiniBand) to a server in the Decode Pool.
  4. Decode Execution: The Decode Pool takes over, grouping this request with dozens or hundreds of other active requests to generate the remaining tokens efficiently.

Key Benefits of Disaggregation

Separating these workloads onto distinct hardware pools provides immediate architectural advantages for large-scale operators:

  • Independent Scaling: Traffic patterns vary wildly by application. A document summarization tool has massive prefill demands and short decode outputs, while a creative writing assistant has short prompts and long decode outputs. Disaggregation allows engineers to scale the ratio of prefill nodes to decode nodes independently based on real-time traffic demands.
  • Hardware Specialization: Operators no longer need to buy one-size-fits-all GPUs. The Prefill Pool can utilize hardware optimized for dense compute (high FLOPS, lower memory bandwidth), while the Decode Pool can utilize hardware optimized for high HBM capacity and massive memory bandwidth.
  • Guaranteed Latency SLAs: Because incoming prefill jobs can no longer interrupt ongoing decode streams, token generation remains smooth and predictable. Operators can independently tune the Prefill Pool to optimize TTFT and the Decode Pool to optimize TPOT.
  • Higher Throughput per Dollar: By eliminating the interference between phases, decode nodes can run at much larger batch sizes without risking latency spikes. This drastically increases the overall token throughput of the cluster, driving down the cost per million tokens generated.