The Transformer Architecture
How attention, a small neural network, residual connections and normalisation stack into the block that every modern LLM repeats dozens of times.
The big picture
A transformer is a neural network built from one repeated unit, the block. Stack 12, 32 or 96 identical-shaped blocks and you have most of today's language models.
The path of a piece of text:
- Split into tokens (pieces of text; see the tokenization lesson) and look up each token's vector in an embedding table.
- Add information about each token's position.
- Pass the vectors through blocks.
- Turn the last vectors into scores over the vocabulary, one score per possible next token.
1 of 7