Part IV · Ch. 02 — Diffusion Model

draft

*Connectionist · Image (and audio/video) generation · U-Net / DiT + diffusion process.* Generates data by learning to reverse a gradual noising process. Card: ../02-types-of-ai/02-connectionist.kmd.

🎨 Figure F-IV.2.0 — From noise to image. Brief: sequence of frames showing pure noise turning into a sharp image in denoising steps; a "forward" arrow (adds noise) and a "reverse" one (removes). Compendium palette.

Diffusion Model

1. Definition and short history

A generative model that learns to invert a chain of Gaussian noise addition. Lineage: DDPM (2020) → latent diffusion/Stable Diffusion (2022) → DiT and flow matching (2023+). It replaced GANs as the state of the art in image generation (see Part III, era 6).

2. Foundations

  • Physics / statistical mechanics — diffusion and Langevin processes inspire

    the method.

  • Probability / stochastic processes — Markov chain of noise; SDEs.
  • Measure theory — continuous formulation (score-based / SDE).
  • Linear algebra and calculus — the score = gradient of the log-density.
  • Information theory — ELBO as the variational objective.

3. Algorithms and architectures

  • Forward — adds Gaussian noise over T steps until pure noise.
  • Reverse — the network predicts the noise (or the score) at each step; denoising.
  • Backbone — U-Net (convolutional) or DiT (Transformer over patches).
  • Latent diffusion — diffusion in the latent space of a VAE (much cheaper).
  • Conditioning — text via cross-attention (CLIP/T5); *classifier-free

    guidance*; ControlNet (poses, edges).

  • Accelerated sampling — DDIM, solvers (DPM-Solver), distillation (turbo).

4. Inputs

  • Hardware: GPU (training and inference); inference lighter than an LLM but

    multi-step.

  • Data: web-scale image-text pairs (LAION-like), filtered.
  • Data structures: image tensors, VAE latents, text embeddings.
  • Systems: PyTorch/diffusers; noise schedulers; guidance pipelines.

5. Specialized life cycle

Stage Specialization
0 Problem Define target generation (imageaudiovideo), conditioning (text?), quality
1 Data Image-text pairs, deduplication, filtering (NSFW, quality, aesthetics)
2 EDA Distribution of resolutions, captions, dataset biases
3 Modeling U-Net vs DiT, latent space (VAE), scheduler, conditioning
4 Training Predict noise/score; checkpointing; long and costly
5 Evaluation FID, CLIP-score, human evaluation of fidelity and prompt alignment
5.5 Acceptance Safety filters (content), red team of prompts, watermark
6 Production Sampling in N steps, guidance, acceleration (DDIM/distillation), queues
7 Monitoring Quality, latency per step, abuse/forbidden content
8 Retraining New data/styles, fine-tuning (LoRA, DreamBooth)
9 Governance Provenance, style copyright, watermark, consent

6. Capabilities, modes and modalities

Strong in artistic/visual mode: text→image, inpainting, outpainting, super-resolution, editing; extends to audio, music and video. Creativity = conditioned sampling ([metaphor], not "imagination").

7. Limits, risks and ethics

Multi-step cost; deepfakes and disinformation; aesthetic/representation biases; style copyright; harmful content. Mitigations: filters, watermarking (C2PA), data curation.

8. State of the art and examples

Stable Diffusion 3, DALL·E, Midjourney, Imagen, FLUX; trend: flow matching, DiT, video generation (see ch. 10), distillation for real time.