Multimodal Models
See how images become tokens for a transformer (ViT), how CLIP aligns pictures and text, and how vision-language models combine them with an LLM.
What multimodal means
A modality is a type of data: text, image, audio, video. A multimodal model reads or writes more than one. The practical question is how to feed non-text data into a transformer, which only consumes a sequence of vectors.
The recipe repeats across modalities: cut the input into pieces, turn each piece into a vector, and treat the vectors like tokens.
| Piece | How it becomes a token |
|---|---|
| Text | Subword, looked up in an embedding table |
| Image | A square patch of pixels, flattened and projected |
| Audio | A short slice of a spectrogram (a picture of frequencies over time) |
| Video | Patches from several frames |
1 of 6