Non-NVIDIA GPUs and ASICs for AI

AMD — Instinct Line (Data Center)

MI300X (2024)

• Differentiator: Largest VRAM per GPU on the market until B300
• Large models: Llama 70B fits in 1 GPU without sharding
• Software: ROCm 6.x; PyTorchCUDA compatible via HIP; vLLMSGLang support
• Adoption: Microsoft Azure, Google Cloud, Oracle Cloud

MI355X (2025)

• Direct competitor to the H200/B100
• AMD ROCm 6.3+ software

MI450 (2026)

• HBM4: bandwidth 3.7× higher than MI300X
• Designed to compete with GB200

---

Google — TPU (Tensor Processing Unit)

TPU v4 (2021)

• Chips per pod: 4,096
• Compute: 275 TFLOPS BF16 per chip
• Interconnect: 3D torus ICI (Inter-Chip Interconnect)
• Use: PaLM training, Gemini 1.0

TPU v5e (2023)

• Focus: Efficiency; lower cost per token
• Configuration: 256 chips per slice
• Compute: 197 TFLOPS BF16 per chip
• Use: Gemini 1.5 Pro inference; cost-efficiency workloads

TPU v5p (2023)

• Focus: Maximum training performance
• Compute: 459 TFLOPS BF16 per chip
• Configuration: 8,960 chips in the largest pod
• Use: Training of the frontier Gemini models

TPU v6 "Trillium" (2024)

• Compute: 918 TFLOPS BF16 per chip (~4.7× vs v4)
• HBM: 32 GB HBM2e per chip
• ICI: 1.2 TB/s total
• Use: Gemini 2.x training and inference

TPU v7 "Ironwood" (Nov/2025)

• Compute: 4,614 TFLOPS per chip — analysts: "on par with Blackwell"
• Use: Gemini 3 training
• Comparison: Considered pari passu with NVIDIA's GB200 on Google-optimized workloads

TPU 8t / TPU 8i (Apr/2026 — Google Cloud Next)

• TPU 8t: Optimized for model training
• TPU 8i: Optimized for model inference (new product)
• Context: Launched at Google Cloud Next 2026 to reinforce competition with NVIDIA

Market note: Anthropic signed the largest TPU contract in Google's history — hundreds of thousands of Trillium TPUs in 2026, scaling to 1M by 2027.

Access: Google Cloud (TPU VMs, Google Kubernetes Engine) Frameworks: JAX (native), PyTorch/XLA, TensorFlow

---

AWS — Trainium and Inferentia

Trainium 2 (2024)

• Manufacturer: Amazon (TSMC 3nm)
• Compute: ~840 TFLOPS BF16 per chip
• Configuration: Trn2.48xlarge: 16 Trainium 2 chips
• HBM: 96 GB HBM3e per chip
• Interconnect: NeuronLink v2 (168 GB/s peer-to-peer)
• Cluster: EFA (Elastic Fabric Adapter) for up to 65,536 chips
• Framework: AWS Neuron SDK (PyTorch compatible)

Trainium 3 (Dec/2025)

• Compute: 2.52 PFLOPS FP8 per chip — confirmed by AWS
• HBM: 144 GB HBM3e per chip
• Status: Shipping since December 2025
• Context: Amazon trained Anthropic's models on 500K Trainium 2 chips in its datacenter in Indiana

Inferentia 2 (2023)

• Focus: Low-cost, low-latency inference
• Compute: 190 TFLOPS BF16
• HBM: 32 GB HBM
• Use case: inf2 instances on EC2

---

Intel — Gaudi

Gaudi 3 (2024)

• Software: Intel Gaudi SDK (PyTorch/TensorFlow compatible)
• Price: ~40% cheaper than equivalent H100
• Limitation: Smaller ecosystem; fewer optimized kernels
• Partner: OEM in Dell, HP servers

---

Cerebras — Wafer-Scale Engine

WSE-3 (2024)

• Concept: An entire CPU/GPU fabricated as a single silicon wafer
• Advantage: Zero latency between cores (on-chip SRAM vs HBM)
• Limitation: No HBM; small models very fast; large ones need partitioning
• Use: Training of specific models; scaling research

---

Groq — LPU (Language Processing Unit)

• Architecture: Streaming; deterministic; no KV cache (re-computes)
• Speed: 800 tokens/second with Llama 3 70B (100× faster than a comparable GPU)
• Latency: < 1ms time-to-first-token
• Limitation: Lower throughput in batch; high cost per chip
• Use: Speed demos; latency-sensitive applications
• API: groq.com/api

---

SambaNova — SN40L

• Architecture: Reconfigurable Dataflow Architecture (RDA)
• Innovation: Socket Design — integrated CPU + GPU + memory bank
• On-chip SRAM: 520 MB per RDU
• Advantage: Energy efficiency in inference of large models
• Use: Enterprise; fine-tuned models

---

Graphcore — IPU (Intelligence Processing Unit)

• Difference: Bulk Synchronous Parallel; static computational graph
• Focus: Sparsity; knowledge graphs
• Status (2025): Acquired by SoftBank; uncertain future

---

Inference Comparison — Llama 3 70B

---

Software Frameworks by Hardware