Part IV · Ch. 22 — Classical ML (Trees, Boosting, SVM)

draft

Probabilistic/statistical · Tabular data · Trees / margins / kernels. Statistical models that still beat deep learning on tabular data. Card: ../02-types-of-ai/03-probabilistic.kmd.

🎨 Figure F-IV.22.0 — Forest of decisions. Brief: several decision trees voting (ensemble) on the left; on the right, a maximum-margin hyperplane (SVM) separating classes.

Classical ML — forest and margin

1. Definition and short history

Pre-deep-learning statistical family, still dominant on tabular data. Lineage: SVM (1995), Random Forests (2001), Gradient Boosting (XGBoost, 2014+) — Kaggle champions and production staples in finance.

2. Foundations

  • Statistics / learning theory — bias-variance, generalization (VC).
  • Optimization — boosting (functional descent), maximum margin.
  • Probability — bagging, error estimation.
  • Linear algebra — kernels (SVM).

3. Algorithms and architectures

  • Decision trees — partitions by features.
  • Random Forest — bagging of trees (reduces variance).
  • Gradient Boosting — sequential trees that correct the error (XGBoost,

    LightGBM, CatBoost).

  • SVM — maximum-margin hyperplane + kernel trick.

4. Inputs

  • Hardware: CPU (efficient); optional GPU for large boosting.
  • Data: tabular with engineered features; moderate volumes.
  • Data structures: tables, trees, feature matrices.
  • Systems: scikit-learn, XGBoost/LightGBM.

5. Specialized lifecycle

Stage Specialization
0 Problem Prediction on tabular data, interpretability, low cost
1 Data Tables; feature engineering is decisive
2 EDA Correlations, feature importance, leakage
3 Modeling TreeRFboosting/SVM; regularization
4 Training Fast; cross-validation; hyperparameter tuning
5 Evaluation AUCF1RMSE; feature importance; SHAP
5.5 Acceptance Stability, fairness, robustness
6 Production Light, fast inference; easy to serve
7 Monitoring Feature drift, degradation
8 Retraining Cheap and frequent retraining
9 Governance Explainability (SHAP), bias, regulated decisions (credit)

6. Capabilities, modes and modalities

Predictive/tabular: credit, fraud, churn, pricing, ranking; fast, robust and interpretable — the industry's "workhorse".

7. Limits, risks and ethics

Does not handle unstructured data well (text/image); depends on *feature engineering*; biases in sensitive decisions (credit) require auditing.

8. State of the art and examples

XGBoostLightGBMCatBoost (standard in tabular and Kaggle); still the first choice for tabular data, where it frequently outperforms deep networks.

Complete Probabilistic/Bayesian paradigm (chs. 19–22). Index: INDEX.kmd.