Accelerating AI With Speculative Decoding

Speculative decoding is an optimization technique designed to drastically increase the speed of text generation in large language models (LLMs) without sacrificing output quality. This article explains the fundamental bottleneck of traditional autoregressive generation, details how speculative decoding uses a draft-and-verify mechanism to overcome this limitation, and highlights why this approach yields significant speedups without altering the final text.

The Bottleneck of Autoregressive Generation

Large language models generate text one token (a word or word part) at a time in a process called autoregressive decoding. To generate a five-word sentence, the model must run five separate times. Each step requires loading the model's massive parameters—often tens or hundreds of billions of weights—from the graphics processing unit (GPU) memory into the compute cores.

Because of this, LLM inference is typically "memory-bandwidth bound" rather than "compute bound." The GPU spends far more time moving data around than it does actually calculating the next token. The immense parallel processing power of modern AI hardware sits largely underutilized during single-token generation.

The Draft-and-Verify Mechanism

Speculative decoding solves the memory bottleneck by pairing two models together: a small, fast "draft model" and the large, accurate "target model." The process breaks down into two main phases:

  1. Drafting: The small draft model rapidly generates a speculative sequence of multiple tokens (for example, five tokens in a row). Because this model has very few parameters, it requires minimal memory movement and operates at a fraction of the time it takes the large model to run once.
  2. Verification: The original sequence, plus the newly drafted tokens, are fed into the large target model all at once.

Exploiting Parallel Processing

The core magic of speculative decoding lies in the verification phase. While a large model takes the same amount of time to generate a single token as it does to process multiple tokens in parallel, it can evaluate all the drafted tokens in a single forward pass.

Instead of loading the massive model weights five separate times to generate five tokens, the system loads the weights just once to check the five tokens proposed by the draft model. This shifts the workload from being memory-bound to utilizing the GPU's abundant parallel compute capabilities.

Acceptance, Rejection, and Correction

During the single verification pass, the target model calculates the probabilities for each step in the drafted sequence to see if it agrees with the draft model's choices.

  • If the target model agrees: All drafted tokens are accepted, and the system has successfully generated multiple tokens in the time it normally takes to generate one.
  • If the target model disagrees: The system accepts all valid tokens up to the point of disagreement. The incorrect token and everything drafted after it are discarded. Because the target model has already calculated the correct probability for that failed position during its verification pass, it simply replaces the wrong token with its own correct token.

The process then repeats from the newly established point in the text. Even in the worst-case scenario where the very first drafted token is rejected, the target model still outputs one correct token, meaning the system guarantees progress on every pass.

Lossless Acceleration

A critical advantage of speculative decoding is that it is mathematically lossless. By using a modified rejection sampling method comparing the probabilities of the draft and target models, the final output is guaranteed to follow the exact same probability distribution as if the large target model had generated the text entirely on its own. Users get the speed benefits of a smaller model combined with the intelligence and accuracy of the largest model, resulting in a 2x to 3x speedup in token generation with zero degradation in quality.