Memory and Interconnect for AI

HBM — High Bandwidth Memory

The standard memory for data center GPUs. Stacked vertically (stacking), connected via interposer, it offers much higher bandwidth than GDDR.

HBM Evolution

Why bandwidth matters for LLMs:

• LLM inference is memory-bandwidth bound, not compute bound
• Each generated token needs to load all the weights of the model from memory
• GPT-3 175B: 350 GB of weights → needs 350 GBs for 1 tokensecond
• H100 has 3.35 TBs → allows ~10 tokenssecond in a 2-GPU split

GDDR — Graphics DDR

Memory of consumer GPUs (GeForce RTX, Radeon RX) and some workstation ones.

Disadvantage vs HBM: Lower total bandwidth (single die vs multi-stack); more physical distance → higher latency. Advantage: Much cheaper; more capacity per $.

NVLink — GPU-GPU Interconnect (NVIDIA)

Importance: With NVLink, multiple GPUs act as one larger GPU — model parallelism without PCIe bottleneck.

NVSwitch — Switch for GPU Networks

• Function: All-to-all NVLink among many GPUs (without point-to-point topology restriction)
• NVSwitch 3.0 (Hopper): 8 GPUs full-bandwidth → 900 GB/s all-to-all
• NVSwitch 4.0 (Blackwell): 72 GPUs (GB200 NVL72) with 1.8 TB/s all-to-all
• Analogy: If NVLink is the "highway", NVSwitch is the "toll plaza" that connects them all

InfiniBand — High Performance Network for Clusters

HDR InfiniBand

• Bandwidth: 200 Gb/s per port
• Latency: ~600 ns
• Adoption: Modern H100/A100 clusters

NDR InfiniBand (2022)

• Bandwidth: 400 Gb/s per port
• Latency: ~500 ns
• Switches: NVIDIA Quantum-2 (64 ports)
• Adoption: Standard for new H100 clusters

XDR InfiniBand (2025)

• Bandwidth: 800 Gb/s per port
• Adoption: GB200/B200 clusters

Use: Communication between nodes in distributed training (all-reduce, gradient sync)

RoCE — RDMA over Converged Ethernet

• Alternative: Cheaper InfiniBand using Ethernet
• RoCEv2: Over UDP; widely supported
• Advantage: Existing Ethernet infrastructure; cheaper
• Disadvantage: Higher latency than InfiniBand; more sensitive to congestion
• Adoption: AWS EFA (Elastic Fabric Adapter), Azure RDMA, Google Jupiter

PCIe — CPU-GPU and GPU-GPU Consumer

Limitation for multi-GPU: Without NVLink, two RTX 4090s communicate via PCIe 4.0 x16 = 64 GBs — vs NVLink 4.0 = 900 GBs (14× slower).

NVLink-C2C — Coherent CPU-GPU (Grace Blackwell)

• Bandwidth: 900 GB/s per CPU-GPU pair (bidirectional)
• Coherence: CPU and GPU share a unified address space
• Impact: Eliminates CPU↔GPU data transfer; zero-copy
• Use: GB200 NVL72 — Grace CPU + B200 GPU in the same package

CXL — Compute Express Link (Future)

• Over PCIe 5.0/6.0: Protocol for coherent memory between CPU and accelerators
• CXL 3.0: Allows a shared memory pool across multiple nodes
• Relevance: Allows adding external HBM to the server without a GPU
• Timeline: In adoption in enterprise servers 2025–2027

CPU Memory — DRAM

For ZeRO-Infinity: CPU DRAM as an extension of VRAM — bandwidth is the bottleneck (PCIe 5.0 limits it to 128 GB/s)

NVMe — Storage for AI

• ZeRO-Infinity: Offload of optimizer states to NVMe
• PCIe 5.0 NVMe: 12–14 GB/s read (Samsung 990 Pro, SK Hynix Platinum P41)
• Relevance: Allows training models larger than the available DRAM
• Limitation: 14 GBs vs 100+ GBs of DRAM = 7× slower; only for rarely accessed parameters

Typical Cluster Topology

[Node A]                        [Node B]
GPU0─┐                          GPU0─┐
GPU1─┤ NVSwitch ─── NVLink ─── NVSwitch ─┤GPU1
GPU2─┤   (NVLink 4.0)              │     ├GPU2
GPU3─┘                              └────┘GPU3
 │                                   │
 CPU (PCIe 5.0)              CPU (PCIe 5.0)
 │                                   │
 └──────── InfiniBand NDR ───────────┘
           (400 Gb/s between nodes)

Flash Storage for Datasets

• LLM training: A 15T-token dataset (FineWeb) ≈ 15 TB raw
• Required throughput: 5–10 GB/s to keep the GPU from waiting on data
• Solution: RAID of PCIe 5.0 NVMe or object storage with prefetching (datatrove)
• Cloud: S3GCS with asynchronous dataloader; datasets in sharded ArrowParquet format