How ALiBi Enables Length Extrapolation in AI Models

Attention with Linear Biases (ALiBi) is an innovative positional encoding method used in Transformer models to process sequences much longer than those seen during training. Instead of adding positional embeddings to word tokens at the start of the model, ALiBi applies a static, non-learned penalty directly to the attention scores based on the distance between tokens. This simple adjustment introduces a strong recency bias that degrades distant information predictably, allowing artificial intelligence models to extrapolate to vastly longer context windows without losing performance or requiring extra memory.

Traditional Transformer models have no inherent sense of word order. To fix this, developers usually add sinusoidal or learned positional embeddings to token representations at the bottom of the network. However, these methods fail when a model encounters sequences during inference that are longer than the sequences it saw during training. The model simply does not know how to handle new, unseen position values.

ALiBi completely removes positional embeddings from the input layer. Instead, it intervenes inside the attention mechanism itself. When a Transformer calculates attention, it multiplies a query vector by a key vector to generate a raw attention score, which determines how much focus one word should place on another. ALiBi modifies this process by adding a negative bias directly to this raw attention score before the softmax function is applied.

The bias is strictly proportional to the distance between the query token and the key token. If two tokens are right next to each other, the distance is one, and the penalty is small. If they are ten tokens apart, the penalty is ten times larger. Mathematically, the distance between the two tokens is multiplied by a static scalar value called a slope. Because the bias is negative, words that are further apart receive a lower final attention score, forcing the model to prioritize recent, nearby context.

To allow the model to capture information across different spans of text, ALiBi assigns a different static slope to each attention head in a layer. These slopes are not learned during training; they are fixed before training begins using a geometric sequence. For example, in an eight-head model, the slopes might be fractions like 1/2, 1/4, 1/8, and so on. Attention heads with large slopes penalize distance heavily and focus strictly on local, immediate words. Heads with very small slopes apply a gentle penalty, allowing them to look much further back into the sequence to gather broader context.

This mechanism enables length extrapolation because the penalty scales linearly and remains localized. Because the softmax function exponentiates the attention scores, adding a linear negative bias equates to exponentially decaying the importance of distant tokens. Eventually, when tokens are far enough apart, the penalty becomes so large that their attention scores drop to near zero after the softmax calculation.

When a model trained on 1,000 tokens is suddenly given 10,000 tokens during inference, it does not break. The ALiBi penalties simply silence the tokens that are thousands of steps away, keeping the statistical distribution of the attention scores identical to what the model experienced during training. The model seamlessly processes the massive sequence by focusing on the sliding window of relevant context defined by the static slopes, achieving reliable extrapolation with zero additional trainable parameters.