Reducing AI Hallucinations Using Contrastive Decoding

This article explores how Contrastive Decoding mitigates artificial intelligence hallucinations by pairing a large "expert" language model with a smaller "amateur" model. By subtracting the log probabilities of the amateur model from those of the expert model during text generation, the system isolates and amplifies high-level reasoning while actively penalizing generic, repetitive, and factually incorrect outputs. The following sections break down the mechanics of this dual-model approach, the mathematical subtraction of log probabilities, and the necessity of plausibility constraints to maintain grammatical coherence.

The Flaws of Standard Decoding

Standard language models generate text by predicting the next token (word or sub-word) based on the context provided. Traditional decoding methods, such as greedy search, beam search, or nucleus sampling, rely on a single model's probability distribution. However, even massive, highly trained models assign significant probability to generic phrases, repetitive loops, and plausible-sounding falsehoods—commonly referred to as hallucinations. This happens because standard training objectives reward the model for predicting statistically common surface-level patterns just as much as they reward it for factual accuracy.

The Expert and Amateur Model Dynamic

Contrastive Decoding resolves this issue by introducing two models of different scales from the same model family:

  1. The Expert Model: A large-scale language model (e.g., a 65-billion parameter model) that possesses deep factual knowledge, complex reasoning capabilities, and a strong grasp of nuance.
  2. The Amateur Model: A much smaller language model (e.g., a 1.5-billion parameter model) that lacks deep knowledge and reasoning skills, relying instead on basic grammatical structures, common word associations, and surface-level patterns.

While both models are prone to generating generic text and hallucinations, the amateur model is significantly more likely to make these errors because it lacks the capacity for deep comprehension. Contrastive Decoding exploits this weakness. Because the amateur model's probability distribution represents the "bad habits" of language modeling, it can be used as a filter to clean up the expert model's output.

Subtracting Log Probabilities

At every step of generating a response, both the expert and amateur models evaluate the prompt and output a probability distribution for the next possible token. These probabilities are expressed as log probabilities (the logarithm of the probability), which converts multiplication into addition and subtraction, making mathematical comparisons easier.

Contrastive Decoding calculates a new score for each potential next token using a straightforward formula:

\[\text{Contrastive Score} = \log P_{\text{expert}}(x) - \log P_{\text{amateur}}(x)\]

Here is how this subtraction directly reduces hallucinations:

  • Penalizing Hallucinations and Repetition: If a token is a common hallucination, a clichĂ©, or a repetitive word, both the expert and the amateur model will assign it a high log probability. When the amateur's high score is subtracted from the expert's high score, they cancel each other out, resulting in a low Contrastive Score. The token is effectively suppressed.
  • Amplifying Factual Knowledge: If a token requires deep factual recall or complex reasoning, the expert model will assign it a high log probability. The amateur model, lacking this knowledge, will assign that same token a very low log probability. Subtracting a low (or heavily negative) number from a high number results in a massive boost to the Contrastive Score.

By selecting the token that maximizes this difference, the system forces the AI to generate text that relies strictly on the advanced capabilities of the expert model, stripping away the statistical noise that triggers hallucinations.

The Plausibility Constraint

A raw subtraction of log probabilities introduces a major mathematical risk: division by zero, or in logarithmic terms, subtracting negative infinity.

Sometimes, an amateur model might assign an extremely low probability (near zero) to a token that is entirely nonsensical or grammatically incorrect. If the expert model also assigns a low probability to this nonsense token, but the amateur's probability is slightly lower, the subtraction could accidentally yield a very high Contrastive Score. This is known as a "false positive," and it would cause the system to output gibberish.

To prevent this, Contrastive Decoding applies a plausibility constraint (often referred to as an adaptive plausibility threshold) before doing any math. The system looks exclusively at the expert model's probabilities and filters out any token that does not meet a minimum threshold relative to the expert's top choice.

Once the implausible tokens are discarded, the log probability subtraction is performed only on the remaining pool of valid, high-quality tokens. Because the amateur model is capable of basic grammar, valid functional words (like "the", "is", or "and") easily pass the threshold and are selected naturally when no factual divergence is required. When factual accuracy is required, the subtraction highlights the expert's unique knowledge, resulting in coherent, accurate, and hallucination-free text.