Part IV · Ch. 21 — Gaussian Process (GP)

Probabilistic · Regression/optimization · Kernel methods. Distribution over functions that predicts with calibrated uncertainty. Card: ../02-types-of-ai/03-probabilistic.kmd.

🎨 Figure F-IV.21.0 — Prediction with uncertainty. Brief: mean curve with an uncertainty band that narrows near observed points and widens far from them (confidence band).

1. Definition and short history

Non-parametric Bayesian method that models functions via kernels. Popular in the 2000s-2010s in regression and Bayesian optimization of hyperparameters.

2. Foundations

• Probability — multivariate Gaussian distribution over functions.
• Linear algebra — covariance matrix, inversion (O(n³) cost).
• Statistics — Bayesian inference, marginalization.
• Analysis — kernels (RBF, Matérn) encode smoothness.

3. Algorithms and architectures

• Kernel — defines the similarity/covariance between points.
• Prediction — posterior mean + variance in closed form.
• Bayesian optimization — GP + acquisition function (EI, UCB) to search for

  optima with few evaluations.

• Scaling — sparse approximations (inducing points).

4. Inputs

• Hardware: CPU/GPU; cubic cost in the number of points.
• Data: few points (shines in low-data).
• Data structures: covariance matrix (dense).
• Systems: GPy, GPflow, BoTorch.

5. Specialized lifecycle