StreamingLLM: Infinite Text Generation via Attention Sinks

StreamingLLM is an innovative framework that enables Large Language Models (LLMs) to process and generate infinite-length text streams without sacrificing efficiency or requiring costly retraining. This article explores how StreamingLLM overcomes standard memory and context length limitations by identifying a core model behavior known as "attention sinks." By preserving the Key and Value (KV) states of the first few initial tokens alongside a rolling window of the most recent tokens, StreamingLLM stabilizes the attention mechanism, allowing conversational artificial intelligence to maintain coherent, unending dialogues.

The Challenge of Infinite Streams in Conversational AI

Standard Large Language Models are constrained by a fixed context window determined during their pre-training phase. In conversational AI, this presents two major hurdles during long interactions:

  1. Memory Exhaustion: LLMs store the Keys and Values ( the KV cache) of all previous tokens to avoid recomputing them. As a conversation grows, the KV cache consumes an unsustainable amount of memory, eventually leading to hardware bottlenecks and high latency.
  2. Context Window Limits: Once a conversation exceeds the length of the model's pre-training window, its performance degrades rapidly, resulting in gibberish or lost coherence.

A naive solution to this problem is Window Attention (or a sliding window), where the model only caches the KV states of the most recent tokens and discards everything older. However, in practice, Window Attention fails completely the moment the very first tokens of the conversation are evicted from the cache.

The Discovery of Attention Sinks

Researchers investigating why Window Attention fails discovered an intriguing phenomenon within the Softmax operation used in self-attention mechanisms. The Softmax function requires the attention scores across all available tokens to sum to 1.

Even when the current token being generated does not have a strong semantic match with any previous tokens, the model cannot assign zero attention to everything; the excess attention must be dumped somewhere. LLMs naturally learn to use the very first tokens in a sequence (usually the first one to four tokens) as "attention sinks." Because these initial tokens are visible to almost all subsequent tokens during autoregressive generation, the model reliably dumps unneeded attention weight onto them.

When a standard sliding window discards these initial tokens, the denominator in the Softmax calculation changes drastically. The model loses its anchor, attention scores become erratic, and text generation collapses.

Combining Attention Sinks with a Rolling Window

StreamingLLM solves the sliding window failure by explicitly accommodating the attention sink phenomenon. Instead of maintaining a strictly contiguous window of recent tokens, StreamingLLM divides the KV cache into two distinct parts:

  • The Attention Sinks: The KV states of the first few tokens (typically four) are locked into the cache permanently. They are never evicted, regardless of how long the conversation continues.
  • The Rolling KV Cache: The remaining capacity of the cache operates as a standard sliding window. As new tokens are generated, the oldest tokens after the attention sinks are evicted to make room for the newest ones.

By keeping the initial tokens, the Softmax function retains its required sink to absorb unnecessary attention scores, maintaining normal distribution across the network layers. Meanwhile, the rolling window ensures the model retains the immediate context necessary for coherent responses, while strictly capping memory usage and computation speed.

Positional Encoding Adjustments

Preserving initial tokens while discarding middle ones creates a gap in token positions. For example, a cache might hold tokens 1, 2, 3, and 4, and then jump straight to tokens 1005, 1006, and 1007.

Modern LLMs typically use Rotary Position Embeddings (RoPE) to understand word order. If StreamingLLM used the original absolute positions of the tokens, the model would be confused by the massive gap and the fact that token positions exceed its pre-training limit.

To fix this, StreamingLLM dynamically reassigns positional embeddings within the rolling cache. Instead of encoding tokens based on their absolute position in the infinite stream, positions are assigned based on their current index within the active cache. In the example above, tokens 1005, 1006, and 1007 are dynamically treated as tokens 5, 6, and 7 during the attention calculation. This ensures the relative distances between tokens remain within the limits the model was originally trained to understand, enabling continuous, stable generation indefinitely.