LLM Inference Servers

vLLM

  • Repository: github.comvllm-projectvllm
  • Origin: UC Berkeley (Kwon et al., 2023)
  • Language: Python + CUDA kernels
  • Core technology: PagedAttention + Continuous Batching
  • Support: OpenAI-compatible API; GPTQ, AWQ, GGUF, FP8, INT4
  • Hardware: NVIDIA (primary), AMD ROCm, Google TPU, Intel Gaudi (experimental)
  • Special: Multimodal (image + text), native LoRA serving, prefix caching
  • When to use: Market standard; best ecosystem; general production
  • Throughput: Best throughput per token in production (after SGLang optimizations)

SGLang — Structured Generation Language

  • Repository: github.comsgl-projectsglang
  • Origin: Berkeley Sky Lab (Zheng et al., 2024)
  • arXiv: 2312.07104
  • Core technology: RadixAttention (prefix caching), Compressed Finite State Machine for structured output, CUDA graphs
  • Key advantage: Structured generation (JSON, regex) much faster than vLLM
  • EAGLE-3: Best support for speculative decoding
  • When to use: Workloads with repeated prefixes; structured JSON; agents with tool use
  • Performance: Frequently surpasses vLLM in throughput benchmarks

TGI — Text Generation Inference (HuggingFace)

  • Repository: github.comhuggingfacetext-generation-inference
  • Language: Rust (server) + Python (kernels)
  • Support: Continuous batching, flash attention, speculative decoding, GPTQ, AWQ
  • Hardware: NVIDIA, AMD, Intel Gaudi, Google TPU
  • When to use: Native integration with HuggingFace Hub; HF's Inference API uses TGI
  • API: OpenAI-compatible + HF-native

llama.cpp

  • Repository: github.comggerganovllama.cpp
  • Author: Georgi Gerganov
  • Language: Pure C/C++; no dependencies
  • Format: GGUF
  • Hardware: CPU (x86 AVX2/AVX-512), Apple Silicon (Metal), NVIDIA CUDA, AMD ROCm, Vulkan
  • Advantage: Runs on any hardware; CPU-first; native Q2–Q8 quantization
  • Server: llama-server with OpenAI-compatible API
  • When to use: Local; edge; no GPU; laptops; dev; simplified Docker

Ollama

  • Repository: github.comollamaollama
  • Base: llama.cpp (engine) + model manager
  • Experience: ollama run llama3 — downloads and serves in 1 command
  • Model Hub: Integrated; pull/push models
  • API: REST + OpenAI-compatible (localhost:11434)
  • When to use: Local dev; rapid prototyping; non-technical user; local CI/CD

Jan

  • Repository: github.comjanhqjan
  • Engine: llama.cpp + extensions
  • Interface: Cross-platform desktop app (Electron)
  • API: Embedded OpenAI-compatible server
  • When to use: End user; local graphical interface; privacy

LM Studio

  • URL: lmstudio.ai
  • Engine: llama.cpp
  • Interface: Desktop app (MacWinLinux)
  • When to use: Demos; non-technical users; exploration of local models

TensorRT-LLM (NVIDIA)

  • Repository: github.comNVIDIATensorRT-LLM
  • Base: TensorRT with optimized LLM kernels
  • Support: FP8, INT4, NVFP4 (Blackwell), speculative decoding, paged KV cache
  • Performance: Maximum throughput on NVIDIA hardware
  • Limitations: NVIDIA-only; lengthy compilation; less flexible
  • When to use: Production on NVIDIA (H100B100B200); maximum throughput; enterprise-scale inference

OpenVINO (Intel)

  • URL: docs.openvino.ai
  • Focus: Intel CPU, iGPU, NPU (Neural Processing Unit in Intel Core Ultra laptops)
  • Support: INT8, INT4, open-source models (Llama, Mistral, Qwen)
  • When to use: Edge; servers without an NVIDIA GPU; Intel laptops with NPU

ONNX Runtime

  • URL: onnxruntime.ai (Microsoft)
  • Focus: Cross-platform (CPU, GPU, NPU, iOS, Android, Web)
  • Limitations: Exporting large LLMs is complex; less optimized for generation
  • When to use: Mobile (iOS/Android via React Native); WebAssembly; edge

MLC-LLM (Machine Learning Compilation)

  • Repository: github.commlc-aimlc-llm
  • Origin: CMU (Chen et al., TVM)
  • Mechanism: Compiles models for any hardware via Apache TVM
  • Support: WebGPU (browser!), iOS, Android, CUDA, Metal, ROCm, Vulkan
  • Unique case: The only framework that runs LLMs directly in the browser (WebGPU)
  • When to use: Browser; native mobile; diverse hardware

CTranslate2

  • Repository: github.comOpenNMTCTranslate2
  • Focus: Encoder-decoder models (T5, Whisper, MarianMT) + LLMs
  • Advantage: Very efficient for Whisper (ASR); good for translation models
  • When to use: Whisper for transcription; translation models

Aphrodite Engine

  • Repository: github.comPygmalionAIaphrodite-engine
  • Base: Fork of vLLM focused on roleplay/creative models
  • Extra features: Mirostat sampling, Kobold API, dynamic LoRA
  • When to use: Creative writing; roleplay; Kobold UI

TabbyAPI

  • Repository: github.comtheroyallabtabbyAPI
  • Base: ExLlamaV2 (EXL2 quantization)
  • Advantage: EXL2 is faster than GGUF on NVIDIA GPU; API for TabbyML
  • When to use: NVIDIA GPU with TabbyML for code completion

Inference Servers by Use Case

Scenario Server
NVIDIA production — high throughput vLLM or TensorRT-LLM
Structured output / agents SGLang
HuggingFace Hub integration TGI
Local without GPU llama.cpp / Ollama
End-user desktop Ollama / Jan / LM Studio
Browser / WebGPU MLC-LLM
Mobile (iOS/Android) MLC-LLM or ONNX Runtime
Intel CPU/NPU OpenVINO
Maximum throughput NVIDIA Blackwell TensorRT-LLM with NVFP4

Standard API: OpenAI-Compatible

All modern servers implement OpenAI's API:

  • POST /v1/chat/completions
  • POST /v1/completions
  • GET /v1/models

This makes it possible to swap the backend without changing client code.

Stack for Kode

Environment Server Model
Local dev (laptop) Ollama Qwen2.5-Coder-7B-GGUF Q4KM
Local dev (GPU) vLLM Qwen2.5-Coder-32B AWQ
Server s.r1 (2× RTX 4090) SGLang Qwen2.5-Coder-32B INT4
Production (when H100 available) vLLM + TurboQuant Qwen2.5-Coder-32B FP8