NVIDIA GPUs for AI — Catalog

Data Center Line

H100 (Hopper, 2022–2024)

Variant VRAM Bandwidth FP16 TFLOPS FP8 TFLOPS TDP Interconnect
H100 SXM5 80 GB HBM3 3.35 TB/s 989 1,979 700W NVLink 4.0
H100 PCIe 80 GB HBM3 2.0 TB/s 756 1,513 350W PCIe 5.0
H100 NVL (2-way) 188 GB — — — 600W NVLink
  • Tensor Cores: 4th generation; native FP8
  • NVLink 4.0: 900 GB/s bidirectional across 8 GPUs
  • Use: Standard for training and inference (2022–2025)
  • Availability: AWS p4de, GCP A3, Azure NDv5

H200 (2024)

Variant VRAM Bandwidth FP16 TFLOPS FP8 TFLOPS TDP
H200 SXM 141 GB HBM3e 4.8 TB/s 989 1,979 700W
H200 NVL 141 GB HBM3e 4.8 TB/s 989 1,979 600W
  • Difference vs H100: Only larger and faster memory; same GH100 chip
  • Impact: +40% bandwidth; larger models fit without offload
  • When to use: Inference of 70B+ models without memory sharding

B100 / B200 / B300 (Blackwell, 2025–2026)

Variant VRAM Bandwidth FP16 TFLOPS FP8 TFLOPS FP4 TFLOPS TDP
B100 SXM 192 GB HBM3e 8.0 TB/s 1,800 3,500 7,000 700W
B200 SXM 192 GB HBM3e 8.0 TB/s 2,250 4,500 9,000 1,000W
B300 SXM 288 GB HBM4 15+ TB/s 2,500+ 5,000+ 10,000+ 1,000W
  • Tensor Cores: 5th generation; native FP4 (NVFP4)
  • NVLink 5.0: 1.8 TB/s bidirectional
  • FP4: 2× throughput vs FP8; revolutionary for inference
  • B300: Projected launch Q3 2026 with HBM4

GB200 NVL72 (Grace Blackwell, 2025)

  • Configuration: 36 Grace CPUs + 72 B200 GPUs in a full rack
  • Total VRAM: 72 × 192 GB = 13,824 GB HBM3e
  • GPU-GPU bandwidth: 1.8 TB/s NVLink 5.0 across all 72 B200s (NVSwitch 4.0)
  • CPU-GPU: 900 GB/s NVLink-C2C per CPU-GPU pair
  • Use: Training frontier models (1T+ parameters); inference of giant MoEs
  • Total power: ~120 kW per rack

A100 (Ampere, 2020–2022) — Historical Reference

Variant VRAM Bandwidth FP16 TFLOPS TDP
A100 SXM4 80GB 80 GB HBM2e 2.0 TB/s 312 400W
A100 PCIe 40GB 40 GB HBM2e 1.6 TB/s 312 300W
  • Still in use: Many cloud clusters; cheaper than H100
  • BF16: Introduced in the A100; the training standard ever since
  • NVLink 3.0: 600 GB/s

A40 / A6000 (Ampere — Workstation)

GPU VRAM Bandwidth FP16 TFLOPS
A40 48 GB GDDR6 696 GB/s 149.7
RTX A6000 48 GB GDDR6 768 GB/s 154.8
  • Use: Fine-tuning 13B–34B models on 1 GPU; rendering + AI

Consumer Line (RTX)

RTX 4090 (Ada Lovelace, 2022)

Spec Value
VRAM 24 GB GDDR6X
Bandwidth 1,008 GB/s
FP16 TFLOPS 165.2
INT8 TOPS 661
TDP 450W
Price (launch) ~$1,599
  • Best value for research and consumer fine-tuning
  • Fine-tuning: LLaMA 8B in FP16; 70B models with QLoRA (2–4 GPUs)
  • Inference: Models up to 24B in FP16 or 70B in AWQ INT4
  • NVLink: NOT supported; communication via PCIe (multi-GPU bottleneck)
  • Multi-GPU workaround: Tensor parallelism with PCIe 4.0 (~40% reduction from ideal speedup)

RTX 5090 (Blackwell Consumer, 2025)

Spec Value
VRAM 32 GB GDDR7
Bandwidth 1,790 GB/s
FP16 TFLOPS 838
INT4 TOPS ~3,300
TDP 575W
  • 78% more memory than the RTX 4090; 5× more FP16 throughput
  • GDDR7: Bandwidth nearly 2× that of the 4090
  • Use: Models up to 32B in FP16; fine-tuning 70B models with 2 GPUs

RTX 4080 Super / 4070 Ti Super

GPU VRAM Bandwidth FP16 TFLOPS
RTX 4080 Super 16 GB GDDR6X 736 GB/s 121.9
RTX 4070 Ti Super 16 GB GDDR6X 672 GB/s 79.2
  • Use: Models up to 13B in FP16; Qwen2.5-Coder-7B comfortable

Cost/FLOPS Ratio Comparison (2025)

GPU FP16 TFLOPS Est. price TFLOPS/$
RTX 4090 165 $1,800 91
RTX 5090 838 $2,000 419
H100 PCIe 756 $25,000 30
H200 989 $40,000 25
B200 2,250 $70,000+ 32

The RTX 5090 has the best TFLOPS/$ but no NVLink and no HBM — a memory-bandwidth limit for large models.

Cooling Considerations

GPU TDP Cooling Required
RTX 4090 450W Air cooling (3-slot) or liquid cooling
B200 SXM 1,000W Liquid cooling mandatory
GB200 NVL72 ~120 kW Liquid cooling per rack; rear-door HX

Software Support by GPU

Feature A100 H100 H200 B100/B200 RTX 4090
BF16 Yes Yes Yes Yes Yes
FP8 No Yes Yes Yes No
FP4 (NVFP4) No No No Yes No
NVLink 3.0 4.0 4.0 5.0 No
Transformer Engine No Yes Yes Yes No
FlashAttention 3 Partial Yes Yes Yes Partial