Chinchilla Scaling Laws and the Optimal Token-to-Parameter Ratio

This article provides a quick overview of the Chinchilla scaling laws, which fundamentally altered how researchers train large language models. Published by DeepMind in 2022, the research demonstrated that most existing artificial intelligence models were significantly oversized and undertrained. The study revealed that for compute-constrained AI models, model size and the amount of training data should be scaled in equal proportions. Specifically, the researchers established an optimal ratio of roughly 20 training tokens for every single model parameter to achieve peak performance within a fixed compute budget.

Prior to the Chinchilla paper, the prevailing approach in artificial intelligence was driven by the Kaplan scaling laws, introduced by OpenAI in 2020. The Kaplan laws suggested that the most effective way to improve a language model's performance was to drastically increase the number of parameters while only modestly increasing the training dataset size. This led to the creation of massive models, such as the 175-billion-parameter GPT-3 and the 530-billion-parameter MT-NLG, which were trained on approximately 300 billion tokens.

DeepMind challenged this paradigm by training over 400 language models ranging from 70 million to over 16 billion parameters on 5 billion to 500 billion tokens. By systematically varying the model size and the number of training tokens while keeping the compute budget fixed (measured in FLOPs, or floating-point operations), they mapped out the true compute-optimal frontier.

The findings were clear: to maximize performance for a given compute budget, every time the model size (parameters) is doubled, the dataset size (training tokens) must also be doubled.

Through this scaling relationship, the Chinchilla laws defined the specific optimal ratio: 20 tokens per parameter.

According to this ratio, a 10-billion-parameter model requires 200 billion tokens of training data to be compute-optimal. By contrast, a 175-billion-parameter model like GPT-3 would ideally need 3.5 trillion tokens to reach its full potential—more than ten times the data it actually received.

To validate their findings, DeepMind trained a new model named Chinchilla. It used the exact same compute budget as their previous flagship model, Gopher. However, while Gopher had 280 billion parameters and was trained on 300 billion tokens, Chinchilla was reduced to 70 billion parameters and trained on 1.4 trillion tokens.

Despite being four times smaller in parameter count, Chinchilla significantly and uniformly outperformed Gopher, GPT-3, and other massive models across almost every benchmark, including language modeling, reading comprehension, and common-sense reasoning.

The revelation of the 20:1 token-to-parameter ratio shifted the AI industry away from a sheer parameter arms race. Because smaller models require less memory and computational power to run after training, the Chinchilla scaling laws proved that shifting the training budget toward ingesting more data results in models that are not only smarter and more capable, but also vastly cheaper and faster to deploy in the real world.