Attention

Intuition

Take the sentence "The animal didn't cross the street because it was tired." To understand "it", you look back and decide that it means "animal", not "street".

Attention is a mechanism that lets each token (a word or word piece) do the same thing: look at every other token, decide how relevant each one is, and pull in a weighted mix of their information.

• Input: one vector per token. A vector is a list of numbers; an embedding is a vector that represents a token's meaning.
• Output: one new vector per token, now enriched by the tokens it looked at.
• The weights are not fixed rules. They are computed fresh for every input.