Embeddings
Turn words, images or any item into a vector of numbers so that similar things land close together, then compare them with cosine similarity.
The problem: computers need numbers
A model cannot do arithmetic on the word "cat". The first idea is a one-hot vector: a vector as long as the vocabulary, all zeros except a single 1 at the word's position.
- With 50,000 words, every word is a 50,000-long vector.
- Every pair of words is equally far apart. "cat" is no closer to "dog" than to "tractor".
- Nothing about meaning is stored.
An embedding replaces this with a short, dense vector, for example 384 numbers, where distance carries meaning. Similar things get nearby vectors.
| One-hot | Embedding | |
|---|---|---|
| Length | Vocabulary size (tens of thousands) | Fixed, typically 64 to 4096 |
| Values | Single 1, rest 0 | Real numbers, all used |
| Similarity | None | Close vectors mean similar items |
| Learned? | No | Yes, from data |
1 of 7