How Lasso L1 Drives Regression Coefficients to Exact Zero

In linear artificial intelligence models, the Lasso (Least Absolute Shrinkage and Selection Operator) L1 penalty performs automatic feature selection by driving certain regression coefficients to exact zero. This article explores the mathematical and geometric reasons behind this phenomenon. By examining the constant gradient of absolute values and the sharp diamond shape of the L1 constraint region, we will uncover why L1 regularization eliminates irrelevant features entirely, unlike L2 regularization which merely shrinks them.

Understanding the L1 Penalty

Linear models attempt to minimize a loss function, typically the sum of squared errors, to find the best-fitting line or hyperplane for a dataset. Regularization adds a penalty term to this loss function to prevent overfitting by discouraging overly complex models with large coefficients.

The Lasso penalty uses the L1 norm, which is the sum of the absolute values of the coefficients. The objective function for Lasso regression looks like this:

\[\text{Minimize } \sum (y_i - \hat{y}_i)^2 + \lambda \sum |\beta_j|\]

Here, \(\beta_j\) represents the model coefficients, and \(\lambda\) controls the strength of the penalty. The use of the absolute value \(|\beta_j|\) is the exact mechanism that allows coefficients to reach zero.

The Calculus Perspective: Constant Gradients

During the training of a linear AI model, optimization algorithms like gradient descent update the coefficients iteratively to find the minimum loss. The update relies on the derivative (gradient) of the loss function with respect to each coefficient.

To understand why L1 hits zero, compare its derivative to the L2 penalty (Ridge regression), which squares the coefficients (\(\beta_j^2\)):

  • L2 Penalty Derivative: The derivative of \(\beta^2\) is \(2\beta\). As a coefficient \(\beta\) gets closer to zero, its derivative also gets smaller. Consequently, the steps pushing the coefficient toward zero become infinitely small. The coefficient approaches zero asymptotically but almost never reaches it.
  • L1 Penalty Derivative: The derivative of \(|\beta|\) is a constant: \(+1\) (if \(\beta\) is positive) or \(-1\) (if \(\beta\) is negative).

Because the L1 derivative is constant regardless of how small the coefficient becomes, the penalty applies a relentless, uniform force pushing the coefficient toward zero. If the predictive power of a feature is weak, the gradient of the error term will be smaller than the constant push of the L1 penalty (\(\lambda\)). The optimization algorithm will push the coefficient all the way to zero, and keep it there.

The Geometric Perspective: Sharp Corners

Another way to understand L1 feature selection is through geometry. Minimizing the Lasso objective function is mathematically equivalent to minimizing the model's error subject to a constraint that the sum of the absolute values of the coefficients is less than a specific budget.

Imagine a simple model with just two features, \(\beta_1\) and \(\beta_2\):

  1. The Error Contours: The unpenalized loss function forms elliptical bowls (contours) centered around the standard ordinary least squares solution. The goal is to find the lowest point of this bowl that still touches the constraint region.
  2. The L2 Constraint Region: For Ridge regression (\(\beta_1^2 + \beta_2^2 \le t\)), the constraint region is a circle. When the expanding elliptical error contours hit a circle, they almost always touch at a smooth point away from the axes. Both coefficients remain non-zero.
  3. The L1 Constraint Region: For Lasso regression (\(|\beta_1| + |\beta_2| \le t\)), the constraint region is a diamond centered at the origin. Crucially, this diamond has sharp corners that lie directly on the axes, where one of the coefficients is equal to zero.

As the elliptical error contours expand outward to meet the L1 diamond, they are highly likely to hit one of these sharp corners first, simply because the corners stick out toward the ellipses. When the intersection happens at a corner on an axis, the coefficient corresponding to the other axis becomes exact zero.

Automatic Feature Selection in AI

In machine learning, a coefficient of exact zero means the corresponding feature is multiplied by zero and completely removed from the predictive equation.

When dealing with high-dimensional datasets containing thousands of variables, many features are redundant or irrelevant. By utilizing the constant gradient and sharp geometric constraints of the L1 penalty, Lasso automatically strips away noise. This results in sparser, faster, and more interpretable linear AI models without requiring manual feature elimination beforehand.