Part V · 3 — Hardware

draft

The physical substrate. Here the concepts from Part I (GPU, HBM, interconnect) become numbers, models, and prices — what to buy, what to rent, and the bottleneck that defines everything: memory.

🎨 Figure F-V.3 — The build pyramid. Brief: a 5-level pyramid (student → frontier company), each level with a GPU icon and a cost band; alongside, the rule "if it doesn't fit in VRAM, it doesn't run". Compendium palette.

The build pyramid

3.1 Why GPUs

Neural networks essentially do matrix multiplication. An H100 has ~16,000 CUDA cores + 500+ Tensor Cores → ~2,000 TFLOPS FP16 (≈2,000× a CPU in massive parallelism). The CPU is optimized for sequential logic; the GPU, for parallelism.

3.2 VRAM — the bottleneck that defines everything

Rule: if the model doesn't fit in VRAM, it doesn't run.

Model FP16 INT8 INT4 Where it fits (INT4)
Llama 3.2 3B 7 GB 4 GB 2.5 GB any modern GPU
Qwen2.5-Coder 7B 15 GB 8 GB 5 GB RTX 3060 12GB
Qwen2.5-Coder 32B 66 GB 34 GB 20 GB RTX 3090/4090 24GB
Llama 3.1 70B 140 GB 72 GB 40 GB 2× RTX 3090 / A100 40GB
DeepSeek-V3 671B MoE 1.3 TB 670 GB 350 GB 8× H100 / 4× B200

Training consumes ~16–20 bytes/parameter (full fine-tune). QLoRA (INT4 + LoRA FP16) fits a 70B in ~48GB. With an RTX 4090/5090 (24–32GB): QLoRA up to ~30B, SFT up to ~13B, inference up to ~70B INT4.

3.3 CPU, RAM, NVMe, network

  • CPU: for training, it almost doesn't matter — what matters are the PCIe lanes.

    1 GPU → 8–16 cores; 4+ GPUs → 32–64 cores (Threadripper, 128 lanes).

  • RAM: ~2× total VRAM; DDR5 ECC.
  • NVMe: 7–14 GBs (PCIe 45); a SATA SSD leaves the GPU idle.
  • Network: NVLink (900 GBs) vs PCIe (128 GBs) — critical in multi-GPU;

    multi-node needs 100/400 GbE or InfiniBand.

Level Cost Configuration Capacity
1 — Student ~$1,500 RTX 3090 24GB, Ryzen 7, 64GB QLoRA up to 13B, inference 70B INT4
2 — Workstation ~$5,000 2× RTX 5090 32GB, 128GB QLoRA 70B, training up to 13B full
3 — Lab ~$20,000 4× RTX 6000 Ada, Threadripper, 256GB ECC full fine-tune 30–70B
4 — Startup ~$200k 8× H100 DGX/HGX, InfiniBand training a 70B from scratch
5 — Frontier $10M–100M+ 64–1024 H100/B200, InfiniBand NDR frontier (rent/hire)

Brazil note: hardware is typically 1.8–2.5× more expensive due to taxes.

3.5 Frontier accelerators (2025–2026)

  • NVIDIA Blackwell: B200 (2.5× H100, 25× more efficient), *300/Blackwell

    Ultra; Rubin*(Vera CPU + Rubin GPU) in production H2 2026.

  • AMD: MI355X (4× MI300X), MI450 "Helios" (HBM4, 19.6 TB/s).
  • AWS Trainium3: TSMC 3nm, 2× Trainium2, 40% more efficient.
  • Impact: the H100 dropped ~30% with the arrival of the B200.

3.6 Cloud vs. on-prem

  • RTX 4090 breakeven: 5,500 h of effective use (1.5 year at 10h/day).
  • Rule: start 100% in the cloud to validate traction; buy local if you use it

    ≥6h/day for 6+ months; rent an H100 by the hour for big runs (72h on 8× H100 ≈ $2,000); never buy an H100 without contracted cash flow.

  • Providers: Vast.ai, RunPod (consumer); Lambda, CoreWeave, Nebius

    (datacenter); Modal, Replicate, Together.ai (serverless).

More efficient inference is the topic of doc 04-tools-and-frameworks.kmd (KV cache quantization, speculative decoding: TurboQuant, EAGLE-3).