Part III · Era 6 — Transformer Era (2017–2021)

draft

A single architecture — the Transformer — unifies the field and, combined with scale, produces the first large language models. The "scaling law" becomes strategy.

Narrative

In 2017, the paper "Attention is All You Need" proposes the Transformer: it dispenses with recurrence, parallelizes training and scales like nothing before. Within a few years it dominates language (BERT, GPT), then vision and audio. GPT-3 (2020) shows that merely increasing size and data generates new capabilities (in-context learning) — formalized in the scaling laws. Diffusion models (image generation) emerge and AlphaFold 2 solves protein structure prediction. The frontier comes to be defined by who trains the largest.

Milestones

  • 2017 — Transformer · Vaswani et al. (Google) · self-attention; the

    architecture that underpins almost all of today's frontier AI.

  • 2018 — BERT · Google · bidirectional pretraining revolutionizes NLP.
  • 2018–2019 — GPT / GPT-2 · OpenAI · fluent text generation; debate over

    responsible release.

  • 2020 — GPT-3 · OpenAI · 175 billion parameters; in-context learning.
  • 2020 — Scaling Laws · Kaplan et al. · predictable performance as a function

    of sizedatacompute.

  • 2020 — DDPM (diffusion) · Ho et al. · diffusion models become competitive

    in image generation.

  • 2020 — AlphaFold 2 · DeepMind · solves a 50-year-old problem in structural

    biology.

  • 2021 — CLIP & DALL·E · OpenAI · connect text and image; generation from a

    description.

People

Ashish Vaswani and coauthors — "Attention is All You Need" (2017). The eight authors of the Transformer; several founded AI startups. 🖼️ Photo F-III.16 (illustrate the attention mechanism, if no photo).

OpenAI team (GPT) — scaled up language. GPT-2/3 established the large language model paradigm.

John Jumper & Demis Hassabis — AlphaFold 2. Nobel Prize in Chemistry 2024 for protein structure prediction.

🎨 Figure F-III.17 — The attention that connects everything. Brief: infographic of the self-attention mechanism (a sentence with weight arrows linking each word to all the others), scaling up to a giant graph representing an LLM; alongside, the folding of a protein (AlphaFold) and an image being born from noise (diffusion). The era in which "attention + scale" opens everything up.

Connection to the Compendium: it introduces the Transformer/LLM and diffusion architectures of the connectionist gallery, and turns the Training stage (Part I) into an industrial-scale enterprise (parallelism, scaling laws, checkpointing).