Griffin: Blending Recurrence and Attention for Fast AI

The Griffin architecture represents a major shift in artificial intelligence language modeling by combining gated linear recurrences with local attention to overcome the efficiency bottlenecks of standard Transformers. While traditional large language models struggle with high memory costs and slow processing speeds during long conversations, Griffin achieves high performance using a hybrid approach. This article breaks down the core components of the Griffin model, explaining how its recurrent blocks, gating mechanisms, and local attention work together to process massive sequences of text quickly and accurately.

The Problem with Standard Transformers

To understand why Griffin is necessary, we must look at the limitations of the standard Transformer architecture. Transformers rely on global self-attention, which compares every word in a sequence to every other word.

While this makes the model very smart, it creates a massive computational bottleneck. As a text prompt gets longer, the memory and computing power required to process it grow quadratically. This makes generating long responses slow and expensive.

Gated Linear Recurrences (The RG-LRU)

Griffin solves the memory bottleneck by reviving an older AI concept: Recurrent Neural Networks (RNNs). Historically, RNNs were fast at generating text because they only looked at the current word and a compressed memory of past words. However, they were slow to train and often forgot information over long distances.

Griffin introduces a new recurrent block called the Real-Gated Linear Recurrent Unit (RG-LRU). It updates the RNN concept in two ways:

• Linear Recurrence: By removing certain complex mathematical steps between layers, Griffin allows the model to be trained across multiple computer chips simultaneously, matching the fast training speeds of Transformers.
• Gating Mechanisms: The "gates" act as intelligent filters. They decide what new information is worth keeping in the model's fixed-size memory and what old information can be forgotten. This allows the model to maintain context over massive documents without running out of memory.

The Power of Local Attention

While gated linear recurrences are great at summarizing the big picture into a compact memory, they are not always perfect at recalling exact, small details. This is where the second half of Griffin's architecture comes in: local attention.

Instead of using global attention to look at the entire document at once, Griffin uses a sliding window of local attention. The model only applies attention to a few hundred words right before the current word. This provides two major benefits:

• Exact Recall: The model can precisely copy and paste information or resolve complex grammar within the immediate vicinity.
• Capped Memory Costs: Because the window size is fixed, the computational cost never grows out of control, no matter how long the document gets.

How the Hybrid Architecture Works

Griffin stacks these different layers together like building blocks to get the best of both worlds. Like the standard Transformer, each layer in Griffin uses a Multi-Layer Perceptron (MLP) to process data, but it alternates the mixing mechanisms.

Specifically, Griffin repeats a pattern of two RG-LRU layers followed by one local attention layer.

When text enters the model, the local attention layer captures the exact details of the immediate sentence. Next, the recurrent layers take that detailed information and compress it into the long-term memory state. As the model generates new words, it pulls from the long-term recurrent memory for overall context and uses the local attention window to ensure the immediate phrasing is accurate.

Efficiency and Performance Benefits

By blending these two techniques, Griffin matches the intelligence and accuracy of standard Transformers of the same size, such as Llama-2, while offering vastly superior efficiency.

Because it does not need to store a massive history of every previous word for global attention, Griffin uses significantly less memory. During text generation, Griffin maintains a constant speed and memory footprint, allowing it to effortlessly process sequences tens of thousands of words long at a fraction of the computational cost.