Quantifying AI Sensitivity with Concept Activation Vectors

Testing with Concept Activation Vectors (TCAV) is an interpretability technique that translates the complex, internal states of deep learning models into human-understandable concepts. Rather than relying solely on pixel-level importance, TCAV measures how sensitive an artificial intelligence's final prediction is to high-level ideas, such as "striped textures" when identifying a zebra. This article explains the mechanics of defining a concept, generating the activation vector in an internal neural network layer, and mathematically quantifying the model's sensitivity to that specific concept.

Defining the Human-Understandable Concept

Neural networks operate in high-dimensional mathematical spaces that do not naturally align with human reasoning. To bridge this gap, TCAV requires the user to define a concept using a set of example data.

To test for a "striped texture" concept, a user gathers two distinct sets of data:

Concept Examples: A collection of images that clearly feature stripes (e.g., striped shirts, curtains, or abstract patterns), intentionally excluding images of the target class (like actual zebras) to avoid bias.
Random Examples: A collection of arbitrary images that do not contain stripes.

Generating the Concept Activation Vector (CAV)

Once the datasets are prepared, both the striped images and the random images are passed through the pre-trained neural network up to a specific internal layer of interest. Instead of looking at the final output of the AI, TCAV extracts the numerical activations (the internal representation of the data) at that specific layer for every image.

Next, a simple binary linear classifier is trained to separate the activations of the striped images from the activations of the random images. The Concept Activation Vector (CAV) is defined as the vector orthogonal (perpendicular) to the decision boundary of this linear classifier. In simple terms, the CAV is a mathematical arrow in the neural network's activation space that points directly away from random noise and toward the concept of "stripedness."

Calculating Conceptual Sensitivity with Directional Derivatives

Having a vector that points toward the "striped" concept allows researchers to measure how much that concept influences a specific prediction, such as classifying an image as a zebra. TCAV achieves this using directional derivatives.

For a single image of a zebra, the system calculates the gradient of the model's "zebra" prediction with respect to the internal layer's activations. It then computes the directional derivative by taking the dot product of this gradient and the CAV.

This calculation answers a straightforward question: If the internal activations of this image are nudged slightly further in the direction of the "striped" vector, does the AI's probability of predicting "zebra" increase or decrease? A positive value indicates that the concept of stripes actively encourages the model to classify the image as a zebra.

Aggregating the Final TCAV Score

Sensitivity for a single image is not enough to understand the behavior of the entire layer. To quantify the global sensitivity of the AI to the concept, the directional derivative is calculated across an entire dataset of target images (e.g., hundreds of different zebra photos).

The final TCAV score is the ratio of target images that have a positive directional derivative. For example, if 85 out of 100 zebra images yield a positive derivative when tested against the "striped" CAV, the TCAV score is 0.85. A score near 1.0 proves the internal layer relies heavily on striped textures to make its decision, while a score near 0.5 suggests the concept is irrelevant to the model's output.

Validating with Statistical Significance

Because neural networks operate in high-dimensional spaces, a single CAV might accidentally align with the target class by pure chance. To ensure the quantified sensitivity is valid, TCAV performs statistical significance testing.

Researchers generate multiple CAVs using the same "striped" images paired against dozens of different sets of random images. This creates a distribution of TCAV scores for the concept. These scores are then compared using a two-sided t-test against meaningless CAVs (generated by comparing random images against other random images). If the "striped" TCAV scores are consistently and significantly different from the random baseline, the AI's reliance on the concept is confirmed as statistically robust.