Part I · 2 — The stages in detail

draft

Each of the 11 stages, in the same mold: definition → inputs/outputs → activities → resources → sciences → mathematics. The consolidated matrices are in the next doc (03-sciences-and-mathematics.kmd); here the focus is the narrative of each stage.

🎨 Figure F-I.3 — Icon sheet of the 11 stages. Brief: a set of 11 cohesive line icons (same stroke weight, Compendium palette), one per stage: target (0), crystal/database (1), magnifier+chart (2), network skeleton (3), gear+gradient (4), ruler+note (5), shield+check (5.5), rocketserver (6), sensoroscilloscope (7), circular arrow (8), scale+shield (9). Reused as markers throughout the whole Compendium.

The icons of the 11 life-cycle stages

Stage 0 — Problem definition

Definition. Translate a need (business, scientific, creative) into a well-posed AI task, with a success metric and an objective function.

  • In: need, constraints, context. Out: problem spec +

    metric + acceptance criterion.

  • Activities: choose the task type (classification? regression? ranking?

    generation? control?), define the target metric, map risks and feasibility.

  • Resources: product-framing frameworks; no heavy machinery.
  • Sciences: decision theory, economics (cost-benefit), *hilosophy/

    ethics*(what should be built), psychology (user need), cognitive science (which intelligence capability to address).

  • Mathematics: logic (formalizing the objective), combinatorics/complexity

    (is the problem tractable?), game theory (are there incentives/adversaries?), statistics (choice of metric and baseline).

It is the least "mathematical" and most conceptual stage — and the one that most dooms projects when poorly done. A badly framed problem is not fixed with more GPU.

Stage 1 — Data collection and engineering

Definition. Acquire, label, clean, and transform the data that will be the AI's "knowledge".

  • In: problem spec. Out: clean, versioned dataset, with features.
  • Activities: acquisition (scraping, sensors, databases), labeling, cleaning,

    feature engineering, deduplication, balancing, splits (trainvaltest).

  • Resources:
    • Hardware: CPU (ETL), massive storage (data lakes, object

      storage, NVMe), network.

    • Algorithms: sampling, hashing/dedup, outlier detection, tokenization

      (BPE for text), feature extraction by modality.

    • Data structures: tensors, columnar tables (Parquet/Arrow),

      feature stores (B-trees/LSM-trees), embeddings tables.

    • Systems: pipelines (Spark, Airflow, Kafka for streaming), data

      versioning (DVC).

  • Sciences: statistics (sampling, selection bias), linguistics

    (text corpora), acousticsopticsDSP (audio/image signals), sociology (social bias in data), law (privacy, licensing).

  • Mathematics: sampling and descriptive statistics, probability (bias,

    representativeness), linear algebra (feature transformations), information theory (redundancy, compression).

A maxim of the field: "garbage in, garbage out". Data quality is the ceiling of model quality — hence the data-centric AI movement.

Stage 2 — Exploratory analysis (EDA)

Definition. Understand the real behavior of the data before modeling.

  • In: dataset. Out: insights (distributions, correlations, anomalies)

    that guide the modeling.

  • Activities: visualization, descriptive statistics, correlations, detection of

    outliers and leakage, balance and bias checks.

  • Resources: CPU; analysis libraries (pandas, polars); tabular structures

    and indexes; notebooks.

  • Sciences: statistics (the protagonist), psychometrics (if there are

    human labels), sociology (bias), the domain science (medicine, finance…).

  • Mathematics: inferential statistics, correlation/covariance, *ypothesis

    tests, dimensionality reduction (PCA* t-SNE/UMAP), information theory (entropy, mutual information).

Stage 3 — Modeling

Definition. Decide the shape of the intelligence: the architecture, the formulation of the objective, the inductive biases (priors) built in.

  • In: EDA insights. Out: defined architecture + loss function +

    initial hyperparameters.

  • Activities: choose the model family, design the architecture, define the

    loss, embed symmetries/priors, size capacity (parameters × data).

  • Resources:
    • Algorithms/architectures: Transformers, CNNs, diffusion, GANs/VAEs,

      RNN/LSTM, GNNs, MoE, SSMs (Mamba) — the choice defines almost everything.

    • Data structures: tensors, computational graph (DAG),

      embeddings, sparse tensors (MoE/sparse attention).

    • Systems: frameworks (PyTorch, JAX), compilers (XLA, Triton).
  • Sciences: neuroscience (inspiration of the neuron, attention, memory),

    physics (statistical mechanics → diffusion; Hopfield networks), cognitive science (reasoning architectures), biology/evolution (neuroevolution), linguistics (structure for NLP).

  • Mathematics: linear algebra (representations, attention as a product of

    matrices), group theory/symmetries (equivariance: CNN↔translation), geometry/topology (manifold hypothesis, latent spaces), information theory (objectives as KL/entropy), measure theory (continuous models).

It is the stage where the inspirations from neuroscience and physics weigh most — it is where the architecture ideas come from. It is also where the potential "intelligence" is decided: the architecture defines the hypothesis space that the model can represent.

Stage 4 — Training

Definition. Adjust the model's parameters to the data, minimizing the loss.

  • In: architecture + dataset. Out: trained weights (+ checkpoints).
  • Activities: forward/backward pass, iterative optimization, learning rate

    scheduling, regularization, parallelization across many accelerators, checkpointing; post-training (SFT, RLHF/DPO, distillation).

  • Resources:
    • Hardware: GPU/TPU (Tensor Cores, systolic arrays) — the core;

      HBM memory (the real bottleneck, not FLOPs); interconnect (NVLink, InfiniBand, all-reduce); storage for checkpoints; power/cooling.

    • Algorithms: backpropagation, SGDAdamAdamW, schedulers,

      gradient clipping, LoRA/QLoRA, quantization-aware training.

    • Data structures: tensors, autodiff graph, sparse tensors (MoE).
    • Systems: datatensorpipeline parallelism, FSDP/ZeRO (sharding),

      orchestration (Ray, Slurm, Kubernetes), mixed precision (bf16/fp16).

  • Sciences: electrical eng. (accelerators), materials science

    (semiconductors), physics (thermodynamics/energy), learning psychology (conditioning → reinforcement), neuroscience (plasticity → weight updates).

  • Mathematics: multivariable calculus (gradients, Jacobians, Hessians;

    chain rule = backprop), optimization (the heart), stochastic processes (SGD as SDE, dropout), numerical linear algebra (GEMMs, stability), learning theory (bias-variance, generalization), information theory (cross-entropy/KL as loss).

The most hardware-intensive and continuous-mathematics-intensive stage. Where "training" costs from cents to hundreds of millions of dollars depending on scale.

Stage 5 — Model evaluation

Definition. Measure the statistical quality of the model on data it has not seen.

  • In: trained weights. Out: metrics report + comparison with

    baselines.

  • Activities: measure on the test set, compare models, check fairness,

    robustness and calibration, initial red teaming.

  • Resources: GPU/CPU for evaluation inference; public and

    private benchmarks; results data structures; dashboards.

  • Sciences: psychometrics (test theory — the basis of benchmarks),

    statistics, cognitive science (evaluating reasoning), sociology/ethics (bias and fairness).

  • Mathematics: inferential statistics (confidence intervals,

    significance), confusion matrix, ROC/AUC curves (integration), decision theory (thresholds), fairness metrics.

Crucial distinction: here the model is evaluated. The system around it is evaluated in stage 5.5. Confusing the two is a common error.

Stage 5.5 — Testing / Validation

Definition. Validate the whole system — code, API, integration, infra — before releasing it to the real world. It is software QA + business acceptance.

  • In: evaluated model. Out: validated build, fit for production.
  • Activities: unit/integration tests of the pipeline; API

    contract/schema tests; validation in staging and *shadow* (real traffic without affecting the user); UAT (acceptance by stakeholders); adversarial and robustness tests; loadstress tests; regression; biascompliance checks.

  • Resources: staging environments; load testers; testing frameworks; CI/CD;

    feature flags; canary/shadow infra.

  • Sciences: psychology/UX (UAT, usability), law (compliance),

    software engineering, ergonomics.

  • Mathematics: inferential statistics (power, sample size for

    A/B and canary), sequential statistics (early stopping without inflating type I error), logic (specification and verification), combinatorics (coverage of cases), queueing theory (load tests), cryptography (security tests).

The 5.5 → 6 transition is gradual: canary release → A/B in production → rollback if it degrades. It is not an "on/off".

Stage 6 — Production / Deployment

Definition. Serve inferences efficiently, robustly, and at scale.

  • In: validated build. Out: service in production answering

    requests.

  • Activities: serving, dynamic batching, caching (KV-cache),

    quantization, autoscaling, routing (MoE), load balancing.

  • Resources:
    • Hardware: inference GPUTPUNPU, memory/HBM, network; edge for

      on-device.

    • Algorithms: sampling (top-k/p, temperature), speculative decoding,

      paged attention (vLLM), quantization (int8/int4), pruning, LoRA.

    • Data structures: KV-cache, vector indexes (HNSW) for RAG,

      ring buffers