Decoding AI with Layer-wise Relevance Propagation

This article provides a clear breakdown of how Layer-wise Relevance Propagation (LRP) explains artificial intelligence decisions. We will explore the mechanics of how LRP takes a neural network's final prediction score and redistributes it backward, layer by layer, to identify which original input features contributed most to the outcome. Finally, we will examine how these redistributed scores are converted into visual heatmaps that make complex deep learning models transparent and understandable.

The Forward Pass and Initial Score

Before LRP can work, the neural network must make a standard prediction. Data, such as an image or a block of text, enters the network and passes forward through various hidden layers of artificial neurons. At the end of the network, the model outputs a prediction score. For example, if a network is analyzing an image of a cat, the output layer generates a high numerical value representing the confidence that a "cat" is present. LRP takes this exact output value as the starting point, referring to it as the initial "relevance score."

The Backward Redistribution Process

Once the prediction score is established, LRP reverses the flow of the network. Instead of pushing data forward, it pushes the relevance score backward toward the input layer.

To get from the output layer to the layer just before it, LRP looks at the connections (weights) and the activation levels of the neurons used during the forward pass. The fundamental rule of LRP is simple: neurons that contributed more to the activation of the next layer receive a larger share of the relevance score. If a neuron was inactive or had a weak connection during the forward pass, it receives little to no relevance during the backward pass. This process repeats layer by layer until the relevance scores reach the original input.

The Conservation Property

The mathematical backbone of LRP is the conservation property, which acts like a law of physics for neural networks. As the relevance score is pushed backward from one layer to the next, no relevance is created or destroyed.

The sum of all relevance scores assigned to the neurons in any given layer must equal the original prediction score at the output. If the model outputs a score of 100, the relevance distributed across the preceding layer must add up to 100, and the relevance eventually assigned to the input pixels must also add up to 100. This strict conservation ensures that the final explanation is an exact accounting of the model's decision.

Visualizing the AI Decision

When the backward propagation finally reaches the input layer, the original prediction score has been completely divided among the individual input features, such as the pixels of an image or the words in a sentence.

Each pixel or word now holds a specific numerical value indicating its exact contribution to the AI's final decision. To visualize this, these values are mapped onto a color scale to generate a heatmap:

• Positive Relevance: Input features that actively supported the AI's prediction (e.g., the ears and whiskers of a cat) are typically highlighted in warm colors like red or orange.
• Zero Relevance: Neutral features that had no impact on the decision (e.g., an empty background) remain uncolored or grey.
• Negative Relevance: Features that contradicted the prediction (e.g., a dog's collar in the same image) are highlighted in cool colors like blue.

Through this systematic backward redistribution, LRP turns an opaque mathematical prediction into an intuitive visual map, showing humans precisely what the artificial intelligence "saw" to make its decision.