AI Extensions in x86 CPUs

For about a decade, x86 CPUs have been accumulating ISA extensions dedicated to accelerating the central building block of neural networks — matrix multiplication in reduced precisions (INT8, BF16, FP8). The timeline below covers the extensions that an inference runtime (vLLM, llama.cpp, ONNX Runtime, oneDNN) detects and exploits when running off-GPU.

AVX-512 VNNI (Vector Neural Network Instructions)

  • Vendor: Intel (Cascade Lake, 2019); AMD (Zen 4, 2022).
  • Instructions: VPDPBUSD (INT8 dot-product+accumulate into INT32),

    VPDPWSSD (INT16).

  • Typical gain: 2–4× on INT8 GEMM vs. plain AVX-512 BW.
  • Where it shows up: oneDNN INT8 kernels, llama.cpp Q4/Q8, ONNX Runtime

    CPU EP.

AMX (Advanced Matrix Extensions)

  • Vendor: Intel only; debuts in Sapphire Rapids (Xeon 4th gen,

    2023). Continued in Emerald/Granite Rapids.

  • Model: 8 tiles (2D registers, up to 1 KB each) + a TMUL

    unit performing matrix multiplication in a single instruction.

  • Native formats: INT8, BF16 (FP16 added in Granite Rapids).
  • Typical gain: on the order of 8× over AVX-512 VNNI on dense BF16 GEMM.
  • Status on AMD: absent — one of the motivations for ACE (see below).

AVX10

  • Vendor: Intel (announced 2023); AMD committed support via the EAG (2024).
  • Goal: unify the vector ISA between P-cores and E-cores, ending the

    fragmentation of AVX-512 (which disappeared from Intel consumer chips since Alder Lake).

  • Versions:
    • AVX10.1 (2024): 256-bit baseline + optional 512-bit, same

      instructions as AVX-512.

    • AVX10.2 (2024): new conversions and embedding instructions, foundation

      for ACE.

ACE (AI Computing Extensions)

  • Vendor: AMD + Intel, via the x86 Ecosystem Advisory Group (EAG).
  • White paper: published 04302026.
  • Positioning: "Standard Matrix Acceleration Architecture for x86" —

    the first matrix-acceleration standard common to both manufacturers.

  • Mechanism: an extension of AVX10 with acceleration based on the *uter

    product* instead of per-element FMA.

  • Native formats: INT8, OCP FP8, OCP MXFP8, OCP MXINT8,

    BF16 — aligned with the OCP Microscaling spec.

  • Compute density: 16× vs. the equivalent AVX10 FMA operation

    (same input vectors).

  • Compatibility: reuses existing AVX10 optimizations;

    software enablement underway (compilers, oneDNN, libxsmm, LLVM).

  • Why it matters for local inference: it reduces the friction of moving

    AI workloads to external accelerators. Servers and edge boxes that today run Q4/Q8 in llama.cpp now get, in the same socket, native matrix tiles in OCP formats — the territory Intel covered alone with AMX now becomes common ground for AMD/Intel.

Context — x86 Ecosystem Advisory Group (EAG)

Formed in Oct/2024 by Intel and AMD to standardize the future of x86. It announced four cross-vendor features:

Feature Function
FRED Flexible Return and Event Delivery — IDT replacement for interrupt handling
AVX10 Unified P/E-core vector ISA
ChkTag Memory tagging for hardening (analogous to ARM's MTE)
ACE Matrix acceleration for AI

NVIDIA's CEO (Jensen Huang) stated publicly in 2025/2026 that the alliance was necessary to keep x86 alive in a landscape dominated by ARM in the datacenter (GravitonCobalt) and GPUsTPUs in AI.

Comparative table

Extension Vendor Year Type Typical gain vs. predecessor
AVX-512 VNNI Intel + AMD 2019/2022 INT8 dot-product SIMD 2–4× over AVX-512 BW
AMX Intel 2023 2D tiles + TMUL ~8× over VNNI BF16
AVX10.1/.2 Intel + AMD 2024 Unified vector ISA P/E-core parity
ACE Intel + AMD 2026 Outer-product matrix accel 16× over AVX10 FMA

Implications for the Koder stack

  • Inference routing (policies/sdk-first.kmd, services/ai/kode):

    upon detecting AMX or ACE on the host, CPU runners become viable candidates for small quantized models (Phi-4, Gemma 4 9B, Qwen 7B) without a local GPU — relevant for edge nodes of the Koder Stack.

  • Quantization (05-inference/quantization.md): native support for

    OCP MXFP8/MXINT8 in ACE aligns CPU and GPU on the same inference format, eliminating runtime conversions.

  • Benchmark suite (08-benchmarks/): it's worth adding a

    CPU-only band on Sapphire/Granite Rapids hardware (AMX) and, once ACE is available in real silicon (expected 2027+), redoing the Llama 70B table with a CPU-x86 column.

References