How 1-Bit LLMs Like BitNet Use Ternary Weights

This article explores how 1-bit large language models, specifically the BitNet b1.58 architecture, represent artificial intelligence weights using only the ternary values of -1, 0, and +1. It covers the mathematical quantization process that converts standard high-precision numbers into these three states, explains how this shift eliminates the need for complex multiplication in neural networks, and highlights the massive improvements in memory efficiency and processing speed achieved without sacrificing model performance.

The Shift from 16-Bit to 1.58-Bit

Traditional Large Language Models (LLMs) store their parameters, known as weights, as 16-bit floating-point numbers (FP16 or BF16). These numbers require significant memory and computational power to process.

BitNet b1.58, often referred to as a 1-bit LLM architecture, drastically reduces this requirement. Instead of using continuous decimal numbers like 0.4532 or -1.2049, it restricts every weight in the neural network to one of three possible values: -1, 0, or 1. Mathematically, representing three states requires \(\log_2(3)\) bits, which equals approximately 1.58 bits per weight.

The Quantization Process: Absmean Scaling

BitNet does not simply round existing weights to the nearest whole number. Instead, it trains the model from scratch using a specific quantization method called absolute mean (absmean) scaling.

To convert a standard matrix of weights into ternary values during training, the model performs three straightforward steps:

  1. Calculate the Mean: The model takes a group of weights and calculates the average of their absolute values.
  2. Scale the Weights: Every weight in the matrix is divided by this average value. This centers and scales the weights into a standardized range.
  3. Round and Clip: The scaled values are rounded to the nearest integer and constrained (clipped) so they cannot exceed the boundaries of -1 and +1.

Through this process, large positive weights become +1, large negative weights become -1, and weights that are close to zero become 0.

The Crucial Role of Zero

True 1-bit models only use two values: -1 and +1. However, the addition of the 0 state in BitNet b1.58 is what allows it to match the performance of 16-bit models.

In a neural network, a weight of 0 means that a specific feature or connection is ignored. By including 0, the model gains the ability to perform "feature filtering." It can actively turn off irrelevant pathways, creating a sparse network that is much more accurate and expressive than a strictly binary (-1 or +1) system.

Replacing Multiplication with Addition

The most significant advantage of using -1, 0, and +1 is how it changes the fundamental math inside the computer hardware.

Standard LLMs rely on matrix multiplication. To generate text, the model constantly multiplies inputs (activations) by the 16-bit weights and adds the results together. Floating-point multiplication is energy-intensive and requires substantial silicon space on a microchip.

With ternary weights, multiplication is no longer necessary:

  • If a weight is +1, the input value is simply added.
  • If a weight is -1, the input value is simply subtracted.
  • If a weight is 0, the input value is ignored entirely.

By reducing complex multiplications to simple additions and subtractions, BitNet drastically lowers the energy required for computations. Input activations are still kept at a higher precision (typically 8-bit integers) to maintain accuracy, but the heavy lifting of multiplying matrices is completely removed.

Training with the Straight-Through Estimator

A major challenge in training ternary models is that rounding numbers breaks backpropagation—the method AI uses to learn from its mistakes. Backpropagation requires smooth, continuous numbers to calculate gradients, and rounding creates flat steps where gradients become zero.

To solve this, BitNet uses a technique called the Straight-Through Estimator (STE). During the forward pass (when the model makes a prediction), the weights are quantized to -1, 0, and +1. However, during the backward pass (when the model updates its knowledge), the system bypasses the rounding step and applies updates to a hidden set of high-precision weights. Once the training is completely finished, the high-precision weights are discarded, leaving only the ultra-efficient ternary weights for deployment.