Neural Scaling Laws

Neural scaling laws are empirical relationships that estimate how a model's performance changes as training resources increase. They commonly relate prediction error or loss to model size, training data and computation, often showing smooth improvement with diminishing returns over a measured range.

The scaling relationship

Researchers train models at several scales and fit a mathematical curve between resource use and validation loss. Over an observed range, loss often follows an approximate power law: increasing parameters, data or compute reduces loss predictably, but each additional unit produces a smaller gain.

• On logarithmic axes, the relationship between resource scale and loss is often approximately linear, enabling extrapolation beyond smaller experiments.
• The predicted quantity is usually validation loss, which is a clearer and more continuous measure than performance on a particular benchmark.

Balancing training resources

Model size, dataset size and computation are interdependent. For a fixed compute budget, scaling only the number of parameters may leave the model undertrained, while using too small a model may prevent it from absorbing the available data effectively.

• A compute-optimal plan jointly increases model parameters and high-quality training tokens rather than maximizing either one alone.
• Developers can fit scaling curves from smaller runs to estimate the model size, data volume and computation needed to reach a target loss.
• The curves also help compare whether additional resources should be spent on a larger model, more data or longer training.

Limits and technological relevance

Scaling laws are empirical regularities, not universal physical laws. Their coefficients and exponents depend on model architecture, data distribution, training method and the range over which observations were made.

• Predictions can fail when data quality deteriorates, useful data becomes scarce, architectures change or training enters a new regime.
• Lower loss does not guarantee proportionate gains in every task, nor does it precisely predict the appearance of particular capabilities.
• For indigenous AI development, scaling analysis supports resource planning by revealing whether scarce computing capacity, data or model design is the principal constraint.