Pretraining of LLMs

Scaling Laws

Kaplan et al. (2020) — OpenAI

• arXiv: 2001.08361
• Finding: Power-laws between compute (C), parameters (N), and tokens (D): loss ∝ N^{(-0.076), loss ∝ D}(-0.095)
• Rule of thumb: ~1.7 tokens per parameter (suboptimal — 2020 models were undertrained)

Chinchilla — Hoffmann et al. (2022) — DeepMind

• arXiv: 2203.15556
• Finding: Compute optimality requires scaling model AND data equally: ~20 tokens per parameter
• Example: GPT-3 175B should have trained on 3.5T tokens, not 300B
• Impact: Every pretraining decision from 2022 onward uses Chinchilla as a reference

MFU — Model FLOP Utilization

• Metric: Effective FLOPS / theoretical GPU FLOPS
• Reference: A100 with FlashAttention: 35–45% MFU; target is >45%
• Formula: For dense Transformers: ~6 × N × D FLOPS per token

---

Data and Curation

Data Mixture

• Standard strategy (2024-2026): Web (60-70%) + Code (10-20%) + Books (5%) + Academic (5%) + Other
• FineWeb (HuggingFace): 15T tokens from 96 Common Crawl snapshots; best general open dataset
• FineWeb-Edu: 1.3T tokens filtered by educational quality — best for knowledge/STEM
• Curriculum Learning: Multi-stage (web → high-quality data) improves performance; adopted by OLMo 2, Phi-4

Deduplication

• MinHash LSH: Near-duplicate deduplication via hashing by Jaccard similarity
• Impact: SlimPajama removes 49.6% of RedPajama-V1 through deduplication — improves quality
• Tooling: datasketch, datatrove

Quality Filtering

• Standard filters: Removal of profanity, spam, malicious code, personal data (PII)
• Code quality: Filter by presence of tests, build-passing rate, GitHub star count

---

Parallelism and Infrastructure

Data Parallelism (DP)

• Mechanism: Replicates the model on each GPU; distributes batches; synchronizes via all-reduce
• DDP (PyTorch): Standard implementation; efficient for models that fit on 1 GPU

Tensor Parallelism (TP)

• Mechanism: Distributes individual layer parameters across GPUs
• Required reductions: All-reduce at each layer; communication latency

Sequence Parallelism (SP)

• Mechanism: Splits activations along the sequence dimension
• Combined with TP: Covers dropout, LayerNorm not covered by TP

Pipeline Parallelism (PP)

• Mechanism: Distributes layers across GPUs; micro-batches in a pipeline
• Trade-off: Complexity vs. memory savings in very large models

ZeRO (DeepSpeed)

• ZeRO-1: Partitions optimizer states (~4× memory savings)
• ZeRO-2: + gradients (~8× savings)
• ZeRO-3: + model parameters (~N× savings, N = number of GPUs)
• ZeRO-Infinity: Offload to CPU/NVMe — trains models of any size

FSDP (PyTorch Fully Sharded Data Parallel)

• Alternative to ZeRO: Native in PyTorch; direct integration
• FSDP2 (2024): Implicit/explicit prefetching; tested on 1T-parameter models
• Recommendation: Default for most fine-tuning teams (7B–70B on 2–8 GPUs)

Megatron-LM (NVIDIA)

• GitHub: github.comNVIDIAMegatron-LM
• Supports: TP, PP, DP, Expert Parallelism, Context Parallelism
• Performance: FlashAttention integrated; mixed precision FP16BF16FP8
• When to use: Research and large clusters; steep learning curve

---

Training Optimizations

Gradient Checkpointing

• Mechanism: Saves activations only at checkpoints (e.g., every 10 layers); recomputes on the backward pass
• Trade-off: ~33% more training time; reduces activation memory by 10–20×

Mixed Precision Training

• BF16: Best for training (larger dynamic range than FP16); standard on Ampere+ GPUs
• FP16: Requires loss scaling; more sensitive to overflow
• FP8: Available on H100+; reduces memory and increases throughput; requires care

FlashAttention (see architectures file)

• Mandatory in any modern LLM training

Gradient Clipping

• Standard practice: clip by norm (max_norm=1.0) — prevents gradient explosion

---

Pretraining Stack (April 2026)

---

Pretraining Dataset References

See 04-training/datasets.md for the full catalog.