Part IV · Ch. 33 — Recommendation System
Hybrid · Ranking/personalization · Embeddings + factorization + networks. Predicts preferences and ranks items for each user — the economic engine of the web. Card:
../02-types-of-ai/06-hybrid-neuro-symbolic.kmd.
🎨 Figure
F-IV.33.0— Matching user and item. Brief: two embedding "towers" (user and item) converging toward an affinity score; the resulting ranked feed on the right.
1. Definition and short history
Models user-item affinity to personalize feeds, stores, and streaming. Lineage: collaborative filtering → matrix factorization (Netflix Prize, 2006-09) → deep recommenders and two-tower (2016+).
2. Foundations
- Linear algebra — matrix factorization; embeddings.
- Probability / statistics — preference estimation, implicit feedback.
- Information theory — learning-to-rank, information gain.
- Economics / game theory — ad auctions, incentives, engagement.
3. Algorithms and architectures
- Collaborative filtering — user/item neighborhood.
- Matrix factorization — user and item as latent vectors.
- Two-tower — separate encoders + dot product (retrieval at scale).
- Ranking — gradient boosting / deep networks (DLRM); learning-to-rank.
- Pipeline — candidate generation → ranking → re-ranking (diversity).
4. Inputs
- Hardware: GPU (training/embeddings); CPU + ANN for retrieval.
- Data: interactions (clicks, purchases, watch time), profiles, context.
- Data structures: huge embedding tables, ANN index, feature store.
- Systems: event streaming (Kafka), low-latency online serving.
5. Specialized life cycle
| Stage | Specialization |
|---|---|
| 0 Problem | Objective (engagement, revenue, satisfaction), ethical constraints |
| 1 Data | Interaction logs; implicit feedback; exposure bias |
| 2 EDA | Sparsity, popularity bias, cold start |
| 3 Modeling | Two-tower + ranking; features; multi-task objective |
| 4 Training | Offline training + frequent updates; negative sampling |
| 5 Evaluation | NDCG, recall@k, AUC; offline vs online diverge |
| 5.5 Acceptance | A/B test with significance; content guardrails |
| 6 Production | Candidate generation + ranking in ms; online feature store |
| 7 Monitoring | CTR/engagement, feedback loops, filter bubble, fairness |
| 8 Retraining | Frequent retraining (preferences change fast) |
| 9 Governance | Addiction/engagement, bubbles, transparency, manipulation |
6. Capabilities, modes, and modalities
Predictive/personalization at scale: feeds, e-commerce, streaming, ads, personalized search. Operates over billions of items/users in real time.
7. Limits, risks, and ethics
Feedback loops and filter bubbles; popularity bias; cold start; engagement optimization can harm well-being; manipulation and addiction. Fairness and transparency are central debates.
8. State of the art and examples
YouTubeTikTokNetflix/Amazon recommenders; two-tower + DLRM; trend: generative/LLM models for recommendation, well-being objectives beyond clicks.