Model Quantization

Basic Concepts

Quantization: Reducing the precision of weights (andor activations) from FP32BF16 to smaller integers (INT8, INT4, INT3...).

Precision Bits Memory (7B model) Use
FP32 32 ~28 GB Optimizer states
BF16 16 ~14 GB Standard training
FP16 16 ~14 GB Training (with loss scaling)
FP8 8 ~7 GB H100+; training/inference
INT8 8 ~7 GB Inference; slight loss
INT4 4 ~3.5 GB Efficient inference
INT3 3 ~2.6 GB Experimental
1.58-bit ~1.6 ~1.4 GB BitNet b1.58

PTQ (Post-Training Quantization): Quantizes a trained model — no retraining. QAT (Quantization-Aware Training): Trains while simulating quantization — better quality, higher cost.

PTQ Methods

GPTQ

  • arXiv: 2210.17323 | ICLR 2023
  • Authors: Frantar et al. (IST Austria)
  • Mechanism: Optimizes quantization errors layer by layer using second order (Hessian)
  • Result: INT4 with < 1% perplexity loss vs FP16
  • Speed: ~0.25 GPU-hour to quantize Llama 65B
  • Implementation: auto-gptq, optimum

AWQ — Activation-Aware Weight Quantization

  • arXiv: 2306.00978 | MLSys 2024
  • Authors: Lin et al. (MIT HAN Lab)
  • Mechanism: Identifies "salient" weight channels via activations → scales them to protect precision
  • Result: Better than GPTQ on most benchmarks; faster to apply
  • Implementation: autoawq
  • Adoption: Widely used in models distributed on HuggingFace

GGUF / GGML (llama.cpp)

  • Origin: Georgi Gerganov (llama.cpp)
  • Format: Single file with all weights + metadata
  • Quantizations: Q2K, Q3KM, Q4KM, Q5KM, Q6K, Q8_0
  • Advantage: CPU-friendly; runs on M1/M2 Mac, x86 CPU, without a GPU
  • Use: Ollama, Jan, LM Studio, GPT4All

BitsAndBytes (bitsandbytes)

  • Authors: Tim Dettmers et al.
  • arXiv: 2208.07339 (LLM.int8)
  • Mechanism: INT8 with separation of outliers to FP16; support for NF4 (double quant)
  • Integration: HuggingFace load_in_8bit, load_in_4bit; QLoRA uses NF4
  • Limitation: CUDA-only; performance variable vs GPTQ/AWQ

Extreme Quantization (Sub-4 bits)

BitNet b1.58 (Microsoft)

  • arXiv: 2402.17764
  • Mechanism: All weights in {-1, 0, +1} (1.58 bits); activations in INT8
  • Result: Zero loss in models ≥3B parameters; 71% less memory than BF16
  • Trade-off: Requires training from scratch at 1.58b; PTQ does not work well
  • Implication: Inference with addition operations only — no multiplications

SpQR — Sparse Quantization Representation

  • arXiv: 2306.03078
  • Mechanism: Outliers in FP16; the rest in 3-4 bits; structured sparsity
  • Result: < 1% loss at INT3; better than GPTQ at low precision

QuIP# (Cornell)

  • arXiv: 2402.04396
  • Mechanism: Lattice codebooks for 2-bit quantization; incoherence processing
  • Result: Better quality at 2 bits

AQLM — Additive Quantization for Language Models

  • arXiv: 2401.06118
  • Mechanism: Additive quantization — represents the residual iteratively
  • Result: State of the art at 2 bits (April 2025)

KV Cache Quantization

TurboQuant (Google)

  • arXiv: 2504.19874 | ICLR 2026
  • Mechanism: KV cache at 3.5 effective bits; per-block quantization with activation-based calibration
  • Result: ~6× less memory; 8× speedup on H100; zero quality loss
  • Distinction: Quantizes the cache (runtime activations), not weights

FP8 KV Cache

  • Adoption: vLLM, TensorRT-LLM, SGLang — native FP8 support for the KV cache
  • Result: ~2× cache memory reduction vs FP16

INT4 KV Cache

  • Experimental: May cause degradation in long contexts; active research

NVFP4 (NVIDIA)

  • Hardware: Blackwell (B100B200B300)
  • Mechanism: FP4 with per-block scale factors; native FP4 tensor cores
  • Result: 2× throughput vs FP8; 4× vs FP16
  • Availability: TensorRT-LLM with B100+

Activation Quantization

SmoothQuant

  • arXiv: 2211.10438
  • Mechanism: Moves quantization "difficulty" from activations to weights (scale migration)
  • Result: W8A8 (INT8 weights and activations) with near-zero loss
  • Adoption: TensorRT-LLM, vLLM

QuaRot

  • arXiv: 2404.00456
  • Mechanism: Rotation of weights before quantization to reduce outliers
  • Result: Better than SmoothQuant at INT4

Implementations and Tools

Tool Methods Use
auto-gptq GPTQ GPU; production
autoawq AWQ GPU; production
bitsandbytes INT8, NF4 HuggingFace integrated
llama.cpp GGUF (Q2–Q8) Local CPU/GPU
optimum (HF) GPTQ, BnB, ONNX HF integration
llmcompressor GPTQ, AWQ, AQLM vLLM pipeline
TensorRT-LLM FP8, INT8, INT4, FP4 NVIDIA production
OpenVINO INT8, INT4 Intel; edge

Practical Trade-offs

Scenario Recommendation
1 consumer GPU (24 GB) — 70B model AWQ or GPTQ INT4 + offload
Mac M1M2M3 — CPU GGUF Q4KM or Q5KM
A100 80GB server — throughput FP8 + FP8 KV cache (vLLM)
Edge / mobile INT4 with ONNX Runtime
Maximum quality with quantization AWQ INT4 > GPTQ INT4
Research at extreme bits AQLM or QuIP# (2 bits)

Impact on Kode

  • Local deploy on the dev's laptop: GGUF Q4KM of Qwen2.5-Coder-7B via Ollama
  • Server s.r1: AWQ INT4 of Qwen2.5-Coder-32B (fits in 2× RTX 4090)
  • Fine-tuning with QLoRA: NF4 (bitsandbytes) + LoRA adapters — trains 32B on 1 GPU
  • Production: FP8 when H100 is available; TurboQuant for the KV cache