Local LLM Inference

See 05-inference/servers.md for the complete server reference. This file focuses on the setup and configuration aspects for local use (laptop, workstation, your own server).

Ollama — Simplest Local Experience

# Install (Linux/Mac)
curl -fsSL https://ollama.com/install.sh | sh

# Download and run model
ollama run qwen2.5-coder:7b
ollama run llama3.3:70b

# API server (OpenAI-compatible)
ollama serve  # localhost:11434

# List local models
ollama list

Useful models for dev:

ollama pull qwen2.5-coder:7b      # 4.7 GB
ollama pull qwen2.5-coder:32b     # 20 GB  
ollama pull deepseek-coder-v2:16b # 9.1 GB
ollama pull codellama:34b         # 19 GB

llama.cpp — Maximum Flexibility

# Compile
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j $(nproc) LLAMA_CUDA=1  # NVIDIA GPU
# or: make -j LLAMA_METAL=1    # Apple Silicon

# Download GGUF model
huggingface-cli download \
  Qwen/Qwen2.5-Coder-7B-Instruct-GGUF \
  qwen2.5-coder-7b-instruct-q4_k_m.gguf

# Server
./llama-server \
  --model qwen2.5-coder-7b-instruct-q4_k_m.gguf \
  --ctx-size 8192 \
  --n-gpu-layers 99 \
  --port 8080

vLLM — Local Production Server

pip install vllm

# Start server
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --enable-prefix-caching

# With AWQ quantization (fits on 1 GPU 24GB)
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct-AWQ \
  --quantization awq \
  --gpu-memory-utilization 0.90

SGLang — Structured Output and Agents

pip install sglang

python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-Coder-32B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --enable-torch-compile \
  --enable-radix-cache

Choosing the Quantization Level

For GGUF (CPU or NVIDIAAMDMetal GPU):

Quantization Size (7B) Quality vs FP16 When to use
Q2_K 2.7 GB -15% Very limited RAM
Q3KM 3.3 GB -7% Limited RAM; accepts loss
Q4KM 4.4 GB -3% Recommended default
Q5KM 5.1 GB -1% Best CPU quality
Q6_K 5.9 GB ~0% GPU with extra RAM
Q8_0 7.7 GB ~0% Maximum local quality

Minimum Hardware per Use Case

Model Memory required Hardware
Qwen2.5-Coder-7B Q4 6 GB RTX 3060; M1 8GB
Qwen2.5-Coder-14B Q4 10 GB RTX 3080; M1 16GB
Qwen2.5-Coder-32B Q4 22 GB RTX 4090 (24GB); M2 Ultra
Llama 3.3 70B Q4 48 GB 2× RTX 4090; M2 Ultra 96GB
DeepSeek-R1 671B Q4 420 GB 8× H100 80GB

Integration with Code Editors

Continue.dev (VS Code / JetBrains)

// config.json
{
  "models": [{
    "title": "Kode Local",
    "provider": "ollama",
    "model": "qwen2.5-coder:32b",
    "apiBase": "http://localhost:11434"
  }],
  "tabAutocompleteModel": {
    "title": "Qwen Coder 7B",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b"
  }
}

Cursor / Windsurf

  • Configure an OpenAI-compatible base URL: http://localhost:11434/v1
  • API key: any string (e.g.: "ollama")

Neovim — avante.nvim / llm.nvim

  • Supports OpenAI-compatible API; points to Ollama

Local Performance Profiling

# Benchmark with llama-bench
./llama-bench \
  --model qwen2.5-coder-7b-instruct-q4_k_m.gguf \
  --n-gen 128 \
  --n-prompt 512

# Monitor GPU
watch -n 0.5 nvidia-smi

# Monitor VRAM usage in real time
nvidia-smi dmon -s u -d 1

Docker Compose — Complete Local Stack

services:
  ollama:
    image: ollama/ollama
    ports: ["11434:11434"]
    volumes: ["ollama_data:/root/.ollama"]
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui
    ports: ["3000:8080"]
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    depends_on: [ollama]

  langfuse:
    image: langfuse/langfuse
    ports: ["3001:3000"]
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/langfuse

volumes:
  ollama_data:

Apple Silicon — Ideal Configuration

# MLX (Apple ML framework — faster than llama.cpp on M-series)
pip install mlx-lm

# Run model with MLX
mlx_lm.generate \
  --model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
  --prompt "def fibonacci(n):"

# MLX server
mlx_lm.server \
  --model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
  --port 8080

M3 Max (96GB) performance:

  • Qwen2.5-Coder-32B Q4: ~15 tokens/s
  • Llama 3.3 70B Q4: ~6 tokens/s