LLM Foundations

Introduction to Large Language Models

A large language model learns statistical patterns in token sequences and uses those patterns to predict what token may come next. Repeating that prediction one token at a time produces a longer response.

Give the model the text “The sky is”. Its immediate task is not to write a whole paragraph. It assigns a probability to every token that could come next, such as “blue”, “clear”, or “falling”.

The tokenizer first converts the text into token IDs. An embedding table turns each ID into a vector, and the Transformer updates those vectors by passing them through attention and feed-forward layers.

At the final position, a linear layer produces one score for every vocabulary token. Softmax converts those scores into probabilities. The system selects one token and appends it to the text.

The model then repeats the same process with the longer text. This is autoregressive generation: every new token depends on the tokens already present, so output tokens must be produced in order.