Decoding and Sampling

From scores to words

At each step the model outputs one score per vocabulary token. These raw scores are called logits. Softmax converts them to probabilities that are positive and sum to 1. Then something has to choose one token. That choice is decoding.

Text generation is a loop:

1. Feed the tokens so far into the model.
2. Get probabilities for the next token.
3. Pick one token (the step this lesson is about).
4. Append it, and repeat until an end-of-text token or a length limit.

This is autoregressive generation: each output becomes part of the next input. It also explains why generation is slow: tokens come one at a time.