LLM Serving and Inference Economics

Introduction to LLM Serving and Inference

LLM inference is the work of running a trained model to answer a request. Serving is the production system that queues requests, manages memory and computing resources, streams tokens, and keeps response time and cost within their limits.

A request has two phases. During prefill, the model reads the prompt and prepares the information needed for generation. During decode, it generates new tokens one at a time.

Prefill has enough parallel work to keep the compute units busy. Decode does much less arithmetic per step but must repeatedly move model weights and the growing key-value (KV) cache from memory, so it is usually limited by memory bandwidth.

Do not answer “the model is slow” as if it were one problem. High TTFT points toward queueing or prefill; high TBT points toward the decode loop and its memory traffic.