Part IV · Ch. 06 — VAE (Variational Autoencoder)

draft

Connectionist · Generation / representation · Probabilistic encoder-decoder. Learns a continuous latent space from which it samples new data; today it is a central piece of larger pipelines (latent diffusion, multimodal tokens). Card: ../02-types-of-ai/02-connectionist.kmd.

🎨 Figure F-IV.6.0 — Latent funnel. Brief: an encoder compressing data to a point in a Gaussian latent cloud, and a decoder reconstructing it; sampling a new point generating new data.

VAE — latent space

1. Definition and short history

An autoencoder that regularizes the latent space to be a continuous distribution (2014). Lineage: VAE → VQ-VAE (discrete latent, the basis of tokens) → component of latent diffusion and of multimodal models.

2. Foundations

  • Probability / variational inference — maximizes the ELBO.
  • Information theory — the KL term regularizes the latent; compression.
  • Measure theory — continuous formulation; reparameterization.
  • Linear algebra — encoder/decoder; latent space.

3. Algorithms and architectures

  • Encoder — data → latent distribution (μ, σ).
  • Reparameterization — sample z = μ + σ·ε (allows backprop).
  • Decoder — z → reconstruction.
  • Objective — reconstruction + KL to the prior.
  • Variants — VQ-VAE (discrete codebook), β-VAE (disentanglement).

4. Inputs

  • Hardware: GPU; lightweight compared to LLM/diffusion.
  • Data: the target domain (images, audio).
  • Data structures: tensors; codebook (VQ-VAE).
  • Systems: PyTorch; used as a frontend for other models.

5. Specialized life cycle

Stage Specialization
0 Problem Compression, generation, or latent tokenizer for another model
1 Data Target domain; little labeling (self-supervision)
2 EDA Data structure/variability
3 Modeling VAE vs VQ-VAE; latent dimension; β
4 Training Reconstruction + KL; avoid posterior collapse
5 Evaluation Reconstruction error, sample quality, latent usage
5.5 Acceptance Validate latent as input to diffusion/multimodal
6 Production Encoder/decoder in a pipeline (latent diffusion)
7 Monitoring Reconstruction quality in production
8 Retraining New domains
9 Governance Inherited from the system that uses it

6. Capabilities, modes and modalities

Representation/compression and generation; rarely used alone today — it shines as the latent space of diffusion and as a tokenizer (VQ) for autoregressive multimodal models.

7. Limits, risks and ethics

Samples more "blurred" than GAN/diffusion if used on its own; posterior collapse. Risks inherited from the host pipeline.

8. State of the art and examples

Stable Diffusion's VAE (latent), VQ-VAE/VQGAN, neural audio codecs (EnCodec/Mimi) — ubiquitous as infrastructure, not as an end product.