Vision transformers and CLIP
A vision transformer (ViT) cuts an image into patches and feeds them to a standard transformer as if they were tokens. CLIP trains an image encoder and a text encoder together, so that a picture and its caption get similar vectors.
Why it matters
CNNs needed hand-designed structure for images, and every vision task needed its own labelled data. ViT showed the same transformer recipe works for images given enough data, unifying vision and language architectures. CLIP links the two worlds, so you can search photos with words, classify images with no task-specific training, and give language models eyes.
The idea
A vision transformer (ViT) treats an image like a sentence. Cut the image into small patches, for example 16 × 16 pixels. Turn each patch into a vector, add positions, and run a normal transformer.
CLIP trains an image encoder and a text encoder together, so a picture and its caption land close together in the same embedding space. That lets you search images with text, and classify images with no task-specific training.
ViT reads a photo like a comic strip, panel by panel. CLIP is two translators, one for pictures and one for words, trained until they give the same 'meaning code' to a picture and its description.
Comic panels come in a set order. A ViT has no built-in sense of which patch sits next to which, so it must learn that from position embeddings. That is one reason ViTs need much more data than CNNs.
How it works
- Cut the image into a grid of patches, for example 16 × 16 pixels. A 224 × 224 image gives 14 × 14 = 196 patches.
- Flatten each patch and map it to a d_model vector with one linear layer: the patch embedding.
- Add position embeddings and (often) a special class token, then run ordinary transformer blocks. Read the class token (or the average) for the image's vector.
- CLIP: an image encoder (a ViT) and a text encoder each map their input to a vector of the same size, normalised to length 1.
- Train on huge numbers of image–caption pairs. In a batch of N pairs, compute the N × N matrix of similarities.
- Contrastive loss: each image must pick its own caption out of the N (cross-entropy along each row), and each caption its own image (along each column).
- Zero-shot classification: embed the image, embed texts like "a photo of a dog", and choose the most similar.
Key points
- ViT = patches as tokens + a standard transformer.
- ViTs have fewer built-in assumptions than CNNs, so they need more data (or pretraining) to shine.
- CLIP puts images and text into one shared embedding space.
- Contrastive learning: pull matching pairs together and push mismatches apart.
- CLIP-style encoders power image search and give multimodal LLMs their vision input.
Common mistakes
- Training a ViT from scratch on a small dataset and expecting it to beat a CNN.
- Forgetting position embeddings, so patch order is lost.
- Comparing unnormalised CLIP vectors with a raw dot product instead of cosine similarity.
Not true: CLIP is trained on images that each carry a class label from a fixed list.
In fact: CLIP learns from pictures paired with their captions. No fixed list of classes is needed, which is why it can recognise new categories from a text description.
Not true: A vision transformer trained from scratch on a small dataset beats a CNN.
In fact: A ViT has fewer built-in assumptions about images, so it needs far more data, or pretraining, to do well. On small data a CNN usually wins.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.