Hyena Hierarchy: A Sub-Quadratic Replacement for Attention

This article explores how the Hyena hierarchy acts as a fast, sub-quadratic replacement for the traditional self-attention mechanism in artificial intelligence. By combining implicit long convolutions with data-controlled gating, Hyena breaks the computational bottlenecks of standard Transformers. Below is a clear breakdown of how these two core technologies work together to process ultra-long sequences efficiently without sacrificing model quality.

The Bottleneck of Traditional Attention

To understand the Hyena hierarchy, it is first necessary to look at the flaw in standard Transformers. Traditional self-attention mechanisms calculate the relationship between every single token in a sequence simultaneously. As the sequence length grows, the computational cost grows quadratically. Doubling the input length quadruples the required compute and memory. This quadratic scaling makes processing massive contexts—such as entire books or genomic sequences—prohibitively expensive.

What is the Hyena Hierarchy?

The Hyena hierarchy is an attention-free architecture that reduces this quadratic cost to a sub-quadratic scale. It achieves the core advantages of attention—specifically global context awareness and the ability to dynamically route information based on the input data—by interleaving two distinct mathematical operations: implicit long convolutions and data-controlled gating.

Implicit Long Convolutions

In deep learning, convolutions are typically small filters that slide across data to capture local patterns. To capture the global context of a massive text sequence, a filter must be as long as the input sequence itself.

In traditional convolutional networks, creating a filter that long is impractical because the number of trainable parameters would explode as the sequence gets longer. Hyena solves this using implicit long convolutions.

Instead of learning the massive filter directly, Hyena uses a small neural network (a multi-layer perceptron) to generate the filter dynamically. This decouples the parameter count from the sequence length. Whether the input is a hundred tokens or a million tokens, the model uses a fixed number of parameters to create a filter that spans the entire sequence.

To keep this process fast, Hyena relies on the Fast Fourier Transform (FFT) algorithm. Computing a massive convolution directly is just as slow as standard attention. However, by converting the data into the frequency domain using FFT, the convolution can be calculated in near-linear—or sub-quadratic—time.

Data-Controlled Gating

While long convolutions provide a global view of the data, they are inherently static. Alone, they cannot dynamically choose which specific pieces of information to focus on and which to ignore, a capability that makes traditional attention so powerful.

Hyena introduces data-controlled gating to replicate this dynamic behavior. The mechanism works through the following steps:

  1. Linear Projections: The input sequence is passed through several dense layers to create multiple distinct projections of the data, much like the Query, Key, and Value projections in a Transformer.
  2. Element-Wise Multiplication: Instead of calculating a massive attention matrix, Hyena applies element-wise multiplication (gating) between these projections.
  3. Interleaving: The architecture alternates between applying an implicit long convolution and applying an element-wise gate.

First, the data is projected. Then, a long convolution aggregates global context across the whole sequence. Next, the result is multiplied element-by-element with another projection of the data. This gating acts as an information filter, allowing the model to selectively suppress or amplify specific signals based on the actual content of the input.

Achieving Sub-Quadratic Efficiency

By repeating this interleaved pattern of implicit convolutions and gating, the Hyena hierarchy constructs a deep, data-dependent operator that mimics the reasoning capabilities of self-attention.

Because the element-wise gating scales linearly and the FFT-powered long convolutions scale at \(O(N \log N)\) (where \(N\) is the sequence length), the entire Hyena block remains strictly sub-quadratic. This allows AI models to process context lengths hundreds of times larger than standard Transformers at a fraction of the computational cost.