Part IV · Ch. 04 — Vision Transformer (ViT)

draft

Connectionist · Vision · Transformer. Applies attention to image patches, unifying vision and language under the same architecture. Card: ../02-types-of-ai/02-connectionist.kmd.

🎨 Figure F-IV.4.0 — Image as a sequence of patches. Brief: an image cut into a grid of patches, each patch becoming a token with attention arrows between them; alongside, the same Transformer block from the LLM figure, showing the unification.

Vision Transformer

1. Definition and short history

Treats the image as a sequence of patches and processes it with a Transformer encoder. Lineage: ViT (2020) → DeiT, Swin, DINO (self-supervision), CLIP (text↔image), SAM (segmentation). It became the base of multimodal models.

2. Foundations

  • Linear algebra — attention softmax(QKᵀ/√d)V over patches.
  • Probability / self-supervision — contrastive (CLIP) and masked (MAE)

    pre-training.

  • Linguistics + optics — text-image alignment (CLIP) joins the modalities.
  • Learning theory — ViT needs more data than a CNN (less spatial prior),

    but scales better.

3. Algorithms and architectures

  • Patch embedding — splits the image, projects each patch into a vector + position.
  • Transformer encoder — global attention from the 1st layer.
  • Pre-training — supervised, contrastive (CLIP), or *asked

    autoencoding*(MAE/DINO).

  • Variants — Swin (hierarchical windows), conv-ViT hybrids.

4. Inputs

  • Hardware: GPU; training requires large datasets/compute.
  • Data: images (and image-text pairs for CLIP) at web scale.
  • Data structures: patch tensors, joint text-image embeddings.
  • Systems: same as the Transformer (PyTorch/JAX, parallelism).

5. Specialized life cycle

Stage Specialization
0 Problem Classification, text-image retrieval, segmentation, multimodal base
1 Data Images/image-text pairs (LAION), filtering and deduplication
2 EDA Concept coverage, caption bias, contamination
3 Modeling ViTSwin, patch size, objective (contrastivemasked)
4 Training Pre-training at scale + fine-tune; CLIP uses contrastive loss
5 Evaluation Zero-shot (CLIP), accuracy, retrieval, robustness
5.5 Acceptance Bias tests, robustness, integration as a multimodal encoder
6 Production Feature encoder for search/multimodal; quantization
7 Monitoring Domain drift, retrieval quality
8 Retraining New concepts, domain fine-tuning
9 Governance Representation bias, privacy, surveillance use

6. Capabilities, modes and modalities

Spatial/visual + a bridge to linguistic (CLIP): zero-shot classification, text↔image search, perceptual base of multimodal LLMs (see ch. 14).

7. Limits, risks and ethics

Data-hungry; inherits biases from the captions; same vision risks (surveillance, representation). Less spatial prior than a CNN (needs scale).

8. State of the art and examples

CLIP, DINOv2, SAM, Swin; ViT is today the standard visual encoder of frontier multimodal models.