Serving and Cost
Understand where LLM latency and money go (prefill, decode, KV cache, batching) and the levers that cut both, with back-of-the-envelope maths.
Two numbers to track
For every request there are two costs: time and money. Both come from tokens (the pieces of text a model reads and writes; see the tokenization lesson).
| Metric | Meaning | Who feels it |
|---|---|---|
| TTFT | Time to first token: delay before the answer starts appearing | Every user, in chat |
| Tokens per second | Speed of the stream after it starts | Users reading as it types |
| Total latency | Request start to last token | Batch jobs, tool calls |
| Cost per request | Input tokens x input price + output tokens x output price | Your budget |
1 of 8