AI/ML Paradigms
Conceptual taxonomy of the field. Defines the axes that separate model families and situates the modern LLMs cataloged in the rest of the compendium.
1. Predictive vs Generative
Axis defined by the nature of the output. It is the most visible cut for those outside the field.
| Dimension | Predictive AI | Generative AI |
|---|---|---|
| Question | "What will the outcome be?" | "What would a plausible example look like?" |
| Output | Label, number, probability | Synthesized artifact (text, image, audio, code, molecule) |
| Output space | Small and closed | Huge and open |
| Modeled distribution | P(y|x) |
P(x) or P(x|context) |
| Stochasticity | Generally deterministic | Stochastic by design |
| Typical metric | Accuracy, AUC, RMSE | Perplexity, FID, BLEU, human evaluation |
| Representative models | Regression, XGBoost, ResNet (classification), ARIMA | LLMs, diffusion, GANs, VAEs |
| Applications | Credit scoring, fraud, churn, diagnosis, demand | Chat, code, image, voice, video, molecular design |
Overlap. The boundary is not absolute: an auto-regressive LLM also predicts (the next token). The difference is that it predicts iteratively over an output space so large that the result emerges as creation. Real systems combine both — a RAG uses embeddings + classification (predictive) + generation; a fraud copilot uses a classifier + an LLM that explains the decision.
2. Discriminative vs Generative (formal formulation)
Axis defined by the modeled probability distribution. It is the cut used in textbooks (Bishop §1.5, Murphy ch. 1).
Discriminative models
- Learn
P(y|x)directly — the decision boundary. - Do not model how
xwas generated; only how to separate classes. - Examples: logistic regression, SVM, CRF, classification neural networks.
- Tend to have lower asymptotic error when there is sufficient data (Ng & Jordan, NIPS 2001).
Generative models
- Learn
P(x, y)orP(x)— the joint distribution of the data. - Allow sampling (generating new
x) in addition to classifying. - Examples: Naive Bayes, HMMs, VAEs, GANs, diffusion models, auto-regressive LLMs.
- Converge faster in a low-data regime; can be used in an unsupervised way.
Relationship with the predictive/generative axis
The two axes are not equivalent:
- A discriminative model is always used for prediction.
- A formal generative model can be used both for prediction (via Bayes:
P(y|x) ∝ P(x|y)P(y)) and for synthesis. - What the public calls "generative AI" is the synthetic application of formal generative models — generating new
xfromP(x|context).
3. Supervision regime
Axis defined by the type of training signal.
| Regime | Signal | Example |
|---|---|---|
| Supervised | Human-labeled (x, y) pairs |
ImageNet, spam classification |
| Unsupervised | Only x |
Clustering, PCA, autoencoders |
| Self-supervised | x partitioned to generate a pseudo-label of itself |
Next-token prediction (LLMs), masked language modeling (BERT), contrastive (CLIP) |
| Semi-supervised | Small labeled set + large unlabeled set | FixMatch, pseudo-labeling |
| Reinforcement Learning | Scalar reward from environment/preference | AlphaGo, RLHF, RLVR (DeepSeek-R1) |
Self-supervision is what enabled the current scale: the entire internet becomes a labeled dataset when the label is "the next token." Every modern LLM is trained in this regime, with post-training under SFT (supervised) + RLHF/RLVR (RL) for alignment.
Details in 04-training/pre-training.md and 04-training/post-training.md.
4. Other relevant axes
- Parametric vs non-parametric — the number of parameters is fixed (neural networks, regression) or grows with the data (k-NN, Gaussian processes, kernel methods).
- Frequentist vs Bayesian — point estimates via MLE/MAP vs a full posterior distribution over the parameters.
- Deterministic vs stochastic — identical output for the same input vs sampling (temperature, top-p, diffusion noise).
- White-box vs black-box — interpretable by construction (trees, GLMs) vs requiring post-hoc tools (SHAP, SAEs, circuit tracing — see
10-security/interpretability.md).
5. Where the compendium's models fit
| Family | Predictive/Generative | Discriminative/Generative (formal) | Training regime |
|---|---|---|---|
| Auto-regressive LLMs (GPT, Claude, Gemini, Llama, Qwen, DeepSeek) | Generative | Generative (P(x_t|x_<t)) |
Self-supervised + RLHF/RLVR |
| Encoder-only LLMs (BERT, RoBERTa) | Predictive | Discriminative (after fine-tune) | Self-supervised (MLM) + supervised |
| Diffusion models (Stable Diffusion, FLUX, Sora) | Generative | Generative (score-based) | Self-supervised (denoising) |
| GANs | Generative | Generative (implicit) | Adversarial |
| VAEs | Generative | Generative (latent) | Self-supervised |
| Embeddings (CLIP, E5, BGE) | Predictive (downstream) | Discriminative (contrastive) | Self-supervised |
| AlphaFold, ESM | Predictive (structure) and generative (design) | Hybrid | Supervised + self-supervised |
| Tabular classifiers (XGBoost, LightGBM) | Predictive | Discriminative | Supervised |
| RL models (AlphaGo, game agents) | Policy | — | Reinforcement learning |
6. Timeline milestones by paradigm
Cross-anchored with foundational-papers.md:
- Classical discriminative: Perceptron (1958) → Backprop (1986) → CNNs (1998) → AlexNet (2012) → ResNet (2015) → BERT (2018).
- Formal generative — vision: RBMs (2006) → VAE (2013) → GAN (2014) → DDPM (2020) → Stable Diffusion (2022) → FLUX/Sora 2 (2024–2026).
- Formal generative — language: N-grams → RNNs (1990s) → LSTM (1997) → Seq2Seq (2014) → Transformer (2017) → GPT-234/5.x (2019–2025) → Claude Mythos (2026).
- Reinforcement learning: TD-Gammon (1992) → DQN (2013) → AlphaGo (2016) → AlphaZero (2017) → RLHF (2022) → RLVR / DeepSeek-R1 (2025).
The 2017 inflection (Transformer) + scale (Chinchilla 2022) + RL post-training (RLHF 2022, RLVR 2025) is what fused the axes: the same generative model, trained self-supervised, is used simultaneously for prediction (classification via prompting), generation (chat, code, image), and action (agents with tools).
References
- Ng, A. & Jordan, M. (2001). On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes. NIPS.
- Bishop, C. (2006). Pattern Recognition and Machine Learning, §1.5, §4.
- Murphy, K. (2022). Probabilistic Machine Learning, vol. 1, ch. 1 and 9. https:/robml.github.iopml-book
- Goodfellow, I. et al. (2016). Deep Learning, ch. 5 and 14. https:/ww.deeplearningbook.org/
- LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence — discusses supervision regimes at scale.