LLM Training, Data, and Scaling

Introduction to LLM Training, Data, and Scaling

LLM pretraining learns next-token prediction from a very large text collection. Training design decides which data to use and how to divide limited computing work between model size and the number of training tokens.

Pretraining supplies its own labels: at each position, the tokens on the left are the input and the next token is the target. One document therefore creates many training examples without a person labeling them.

A fixed compute budget can buy a larger model, more training tokens, or some balance of the two. A large model trained on too few tokens is undertrained; adding parameters alone wastes part of the budget.

A compute-optimal plan estimates loss for several model-size and token-count combinations, then chooses the pair predicted to give the lowest loss at the available compute.

Parameter count alone is therefore not a training plan. In the Chinchilla study, a 70-billion-parameter model trained on four times as much data as Gopher outperformed much larger models on most of the reported tasks. The allocation between parameters and tokens mattered, not only the final model size.