Understanding Prefix-Tuning and Prompt-Tuning Mechanics

This article explains how prefix-tuning and prompt-tuning adapt large artificial intelligence models to new tasks by prepending learnable continuous vectors to attention layers while keeping the base model weights completely frozen. You will learn the core mechanics of transformer attention, the specific architectural differences between prompt-tuning and prefix-tuning, and how gradients update only these newly added vectors during the training process.

The Foundation: Frozen Weights and Attention

To understand how continuous vectors steer a model, it is necessary to look at how transformers process information. Transformers rely on the self-attention mechanism, which converts input tokens into three matrices: Queries (\(Q\)), Keys (\(K\)), and Values (\(V\)). The model calculates attention scores by multiplying Queries with Keys, and then uses those scores to weight the Values.

In traditional fine-tuning, error gradients flow backward through the network and update every weight in the model. In both prefix-tuning and prompt-tuning, the base artificial intelligence weights are frozen. During backpropagation, gradients are still calculated to find the path of least error, but the optimizer is prevented from updating the original model parameters. Instead, it only updates a small set of newly introduced parameters—often called "virtual tokens" or "soft prompts"—that exist in continuous vector space rather than corresponding to actual words in the model's vocabulary.

How Prompt-Tuning Works

Prompt-tuning is the simpler of the two methods and operates exclusively at the very first layer of the model (the input embedding layer).

  1. Initialization: A sequence of continuous vectors, matching the hidden dimension size of the base model, is initialized.
  2. Prepending: When an input text is fed into the model, it is first converted into standard token embeddings. The learnable continuous vectors are directly prepended (attached to the front) of these input embeddings.
  3. Forward Pass: This combined sequence flows through the frozen transformer layers. At the first attention layer, the model generates \(Q\), \(K\), and \(V\) matrices for both the prepended vectors and the actual input tokens.
  4. Contextual Shifting: As the data moves up through subsequent layers, the original input tokens attend to the prepended vectors, altering their representations and steering the final output toward the desired task.

Because prompt-tuning only modifies the initial input sequence, it requires the fewest trainable parameters, though it sometimes struggles to match the performance of full fine-tuning on smaller models.

How Prefix-Tuning Works

Prefix-tuning is more expressive than prompt-tuning because it injects learnable continuous vectors into every attention layer of the transformer, rather than just the input layer. Furthermore, it specifically alters the Keys and Values without directly adding to the Queries.

  1. Layer-Wise Injection: At every transformer layer, two sets of learnable prefix vectors are initialized: one for the Keys (\(P_K\)) and one for the Values (\(P_V\)).
  2. Prepending to K and V: During the forward pass at any given layer, the model computes the standard \(Q\), \(K\), and \(V\) matrices from the hidden states of the actual input. The learnable prefix vectors \(P_K\) and \(P_V\) are then concatenated to the front of the standard \(K\) and \(V\) matrices.
  3. Attention Calculation: The Queries (\(Q\)) of the actual input tokens now attend to this newly expanded Key matrix (which consists of both the prefix Keys and the original Keys) and pull information from the expanded Value matrix.
  4. Reparameterization for Stability: Directly optimizing these prefix vectors can lead to unstable training. To fix this, prefix-tuning passes the trainable vectors through a temporary Multilayer Perceptron (MLP) during training to generate the actual \(P_K\) and \(P_V\) matrices. Once training is complete, the MLP is discarded, and only the resulting static prefix vectors are kept for inference.

Summary of the Optimization Process

Both methods achieve task adaptation without modifying base weights through targeted gradient updates. When the model makes a prediction, a loss is calculated based on how far the prediction is from the target. During the backward pass, the calculus chain rule computes gradients all the way back to the prepended continuous vectors. The optimizer updates the numerical values of these vectors to minimize future errors, while the frozen weights of the base AI act merely as a static, complex function through which the signals pass.