Self-Hosted Cluster for LLM Training

──────────────────────────────────────────────────────────────┐
│                     LLM Training Cluster                      │
│                                                               │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐           │
│  │   Node 1    │  │   Node 2    │  │   Node N    │           │
│  │  4× A100    │  │  4× A100    │  │  4× A100    │           │
│  │  2× CPU     │  │  2× CPU     │  │  2× CPU     │           │
│  │  512GB RAM  │  │  512GB RAM  │  │  512GB RAM  │           │
│  ──────┬──────┘  └──────┬──────┘  └──────┬──────┘           │
│         │                │                │                    │
│         └────────────────┼────────────────┘                    │
│                          │                                     │
│                   ┌──────▼──────┐                              │
│                   │  Switch IB  │  ← InfiniBand NDR/XDR        │
│                   │  (32/64 port)│    200/400 Gbps              │
│                   └──────┬──────┘                              │
│                          │                                     │
│                   ┌──────▼──────┐                              │
│                   │  Storage    │  ← Lustre / Ceph / WekaIO    │
│                   │  (100TB+)   │    Parallel filesystem        │
│                   └─────────────┘                              │
│                                                               │
│  ┌─────────────┐                                              │
│  │  Head Node  │  ← Kubernetes master / Slurm controller      │
│  │  (CPU only) │                                              │
│  └─────────────┘                                              │
└──────────────────────────────────────────────────────────────┘

Fat-Tree topology (standard for GPU clusters):

  GPU Nodes          Spine Switches        Leaf Switches
  ┌──┐ ┌──┐          ┌── ┌──┐            ┌──┐ ┌──┐
  │N1│─│N2│──────────│S1│─│S2│────────────│L1│─│L2│
  └──┘ └──          └──┘ └──            └──┘ └──┘
  ┌──┐ ┌──┐          ┌──┐ ┌──┐            ┌──┐ ┌──┐
  │N3│─│N4│──────────│S3│─│S4│────────────│L3│─│L4│
  └──┘ └──┘          └──┘ └──┘            └── └──┘

  Bisection bandwidth: non-blocking (each node has full bandwidth)

/storage/
├── datasets/               ← Training datasets (read-only for workers)
│   ├── code/
│   ├── text/
│   └── multimodal/
├── checkpoints/            ← Training checkpoints (write-heavy)
│   ├── run_001/
│   │   ├── step_1000/
│   │   ├── step_2000/
│   │   └── ...
│   └── run_002/
── logs/                   ← Training logs
├── models/                 ← Final exported models
└── scratch/                ← Temporary data (per job)

# Example of a training job on Kubernetes
apiVersion: batch/v1
kind: Job
metadata:
  name: llm-training-run-001
spec:
  parallelism: 8
  completions: 8
  template:
    spec:
      containers:
      - name: trainer
        image: pytorch/pytorch:2.3.0-cuda12.1
        command: ["python", "train.py", "--config", "llm_7b.yaml"]
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 128Gi
            cpu: "32"
        volumeMounts:
        - name: datasets
          mountPath: /storage/datasets
        - name: checkpoints
          mountPath: /storage/checkpoints
      volumes:
      - name: datasets
        persistentVolumeClaim:
          claimName: datasets-pvc
      - name: checkpoints
        persistentVolumeClaim:
          claimName: checkpoints-pvc
      restartPolicy: Never

#!/bin/bash
#SBATCH --job-name=llm-training
#SBATCH --nodes=8
#SBATCH --gpus-per-node=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=32
#SBATCH --mem=512G
#SBATCH --time=72:00:00
#SBATCH --partition=gpu

srun python train.py --config llm_7b.yaml

Level 1: Operating system
  → Ubuntu 22.04 LTS or Rocky Linux 9

Level 2: Drivers and runtime
  → NVIDIA Driver 550+
  → CUDA 12.4
  → cuDNN 9.x
  → NCCL 2.20+ (GPU-GPU communication)

Level 3: Networking
  → OFED (OpenFabrics Enterprise Distribution)
  → InfiniBand drivers
  → UCX (Unified Communication X)

Level 4: Storage
  → Lustre client or WekaIO agent
  → NFS client

Level 5: Orchestration
  → Kubernetes + Volcano or Slurm

Level 6: ML Framework
  → PyTorch 2.3+
  → DeepSpeed / Megatron-LM / FSDP

Level 7: Monitoring
  → Prometheus + Grafana
  → DCGM (NVIDIA Data Center GPU Manager)
  → Slurm accounting or Kubernetes metrics

import torch.distributed as dist

# Initialize process group with NCCL backend
dist.init_process_group(backend="nccl")

# All-reduce: sum gradients across all GPUs
torch.distributed.all_reduce(tensor, op=dist.ReduceOp.SUM)

# Environment variables to optimize NCCL
export NCCL_PROTO=simple      # simple or ll (low latency)
export NCCL_ALGO=Ring         # Ring, Tree, or CollNet
export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_DISABLE=0      # Use InfiniBand
export NCCL_IB_HCA=mlx5       # HCA driver
export NCCL_IB_TC=106         # Traffic class

Week 1–2: Physical infrastructure
  → Install racks, cables, InfiniBand switches
  → Configure storage (NFS or WekaIO)
  → Install OS on all nodes

Week 3: Base software
  → NVIDIA drivers, CUDA, cuDNN
  → NCCL, UCX, InfiniBand stack
  → Test GPU-GPU communication (NCCL tests)

Week 4: Orchestration
  → Install Kubernetes + Volcano or Slurm
  → Configure GPU scheduling
  → Test distributed job

Week 5: ML stack
  → PyTorch, DeepSpeed/FSDP, Megatron-LM
  → Configure monitoring (DCGM, Prometheus)
  → Test training of a small model

Week 6+: Optimization
  → Tuning of NCCL, I/O, parallelism
  → Throughput benchmark
  → Document procedures

Phase 1: Cloud for training (Lambda Labs / CoreWeave)
  → Rent 8–16 A100 per week
  → Train the model
  → Cost: $5K–$15K per training run

Phase 2: On-premise for inference
  → 2× RTX 4090 for serving
  → Cost: $5K in hardware

Phase 3: Migrate training to on-premise when justified
  → When cloud cost > on-premise cost within 12 months