Understanding DoLa: Contrasting Layers for Truthful AI

This article explores Decoding by Contrasting Layers (DoLa), a novel decoding strategy designed to improve the factual truthfulness of Large Language Models (LLMs) and reduce hallucinations. By contrasting the output predictions (logits) of higher, mature Transformer layers against those of earlier, premature layers, DoLa amplifies factual knowledge stored in the upper levels of the neural network while suppressing purely syntactic and plausible-sounding falsehoods generated in the lower levels. Crucially, this method operates entirely at inference time, enhancing truthfulness without requiring external knowledge retrieval or additional model fine-tuning.

To understand how DoLa works, one must first understand how a Transformer processes information across its layers. An LLM is composed of multiple stacked Transformer layers. As text passes through these layers from bottom to top, the model processes different aspects of language. Research indicates a division of labor within these layers: the lower (earlier) layers focus heavily on syntax, grammar, and basic linguistic structure. Conversely, the higher (later) layers are predominantly responsible for injecting factual, semantic, and world knowledge into the final output.

Hallucinations often occur when the syntactic plausibility from the lower layers overpowers the factual knowledge in the higher layers. Standard decoding methods only look at the logits—the raw, unnormalized predictions for the next word—at the very final layer. If the final layer is still heavily influenced by the lower layers' desire to simply complete a grammatically common phrase, the model may output a fluent lie rather than a hard fact.

DoLa solves this by utilizing an "early exit" mechanism to peek inside the model before it finishes processing. Normally, the hidden state of the final layer is passed through a vocabulary head to generate the final logits. DoLa applies this same vocabulary head to the intermediate, premature layers. This allows the system to see what the model would have predicted at Layer 10, Layer 16, or Layer 20, alongside what it actually predicts at the final layer, such as Layer 32.

Once DoLa has the logits from the final (mature) layer and the logits from an earlier (premature) layer, it contrasts them. Mathematically, this is achieved by taking the log probabilities of the mature layer and subtracting the log probabilities of the premature layer.

This subtraction acts as a factual filter:

• Suppressing Falsehoods: If a word has a high probability in both the early layer and the final layer, it is likely an easy syntactic filler (like "the" or "is") or a common, memorized linguistic trope that may not be factually accurate for the specific prompt. Subtracting the two probabilities reduces the prominence of these generic tokens.
• Amplifying Facts: If a word has a very low probability in the early layer, it means the syntax-focused layers did not see it coming. If that same word suddenly spikes to a high probability in the final layer, it means the upper layers actively injected specific factual knowledge to change the model's trajectory. Contrasting the layers highlights this spike, massively boosting the probability that this factual token will be chosen as the next word.

A critical aspect of DoLa's efficiency is dynamic premature layer selection. Because factual knowledge is injected at different depths depending on the complexity of the prompt, contrasting the final layer against a single, fixed early layer is ineffective. Instead, DoLa groups the earlier layers into candidate buckets. For every single token generated, DoLa calculates the Jensen-Shannon Divergence (JSD)—a statistical measure of how much two probability distributions differ—between the final layer and several candidate early layers.

DoLa dynamically selects the premature layer that has the highest JSD (the greatest difference) compared to the final layer. The logic is straightforward: the point of maximum divergence represents the exact moment before the model injected its factual knowledge. By contrasting the final layer against the specific earlier layer where the predictions differ the most, DoLa maximizes the amplification of the newly introduced factual data.

Finally, to prevent the subtraction process from accidentally boosting tokens that are completely nonsensical, DoLa applies an adaptive plausibility constraint. Before contrasting the layers, it filters out any tokens that do not have a reasonably high probability in the final layer. This ensures that the contrasted logits only rearrange the rankings of plausible words, resulting in outputs that are both grammatically sound and factually truthful.