Open-Source Models — Complete Catalog

Updated in April 2026. Focus on models viable as a base/fine-tuning target for Kode.

---

Llama (Meta)

Llama 4 — Paper: arXiv:2601.11659 (Jan/2026) Llama 4 Behemoth: 288B-A288B (16 experts) — "teacher" model still in training; used for the co-distillation of Scout and Maverick. Links: meta.llama.com · huggingface.co/meta-llama

---

Qwen (Alibaba)

Qwen 3 — Paper: arXiv:2505.09388. Key innovation: thinking budget — adaptively allocates reasoning compute per prompt. Qwen3-VL — Paper: arXiv:2511.21631. Video analysis of up to 2 hours. Qwen3-72B: first open model to surpass GPT-4o on MMLU-Pro. Links: qwenlm.github.io · huggingface.co/Qwen

---

DeepSeek

Architecture: V2V3V3.2 = Multi-Head Latent Attention (MLA) + DeepSeekMoE. V4 replaces MLA with CSA + HCA (hybrid attention) while keeping DeepSeekMoE — the KV cache drops to ~2% of the BF16 GQA8 baseline at 1M context. Links: deepseek.com · huggingface.co/deepseek-ai

DeepSeek-V4 — Technical Details (Apr/2026)

Paper: DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (04242026) PDF: huggingface.codeepseek-aiDeepSeek-V4-Problobmain/DeepSeek_V4.pdf

Released code (MIT):

• V4 inference (model + specific kernels): huggingface.co/deepseek-ai/DeepSeek-V4-Pro/tree/main/inference — the canonical reference cited in §2.3 of the paper. (There is NO github.com/deepseek-ai/DeepSeek-V4 repo — the code lives on HF.)
• Related GitHub repos (github.com/deepseek-ai/): TileKernels (kernel lib in TileLang, updated 04232026 alongside V4) · DeepGEMM (FP8 GEMM kernels, updated 04242026) · FlashMLA (Multi-head Latent Attention kernels — used in V2/V3, kept for compat) · DeepEP (expert-parallel comm).
• Sibling project (non-V4): deepseek-ai/Engram — Conditional Memory via Scalable Lookup (Jan/2026, Apache 2.0). Separate work from DeepSeek; not integrated into V4 (the V4 paper does not cite Engram). Promotional videos have conflated Engram with V4's CSA+HCA — they are distinct things.

Architectural innovations:

• Compressed Sparse Attention (CSA): Compresses every m KV entries into a single entry via pooling with a softmax-gate and a learned positional bias; then applies DeepSeek Sparse Attention (DSA) with a lightning indexer that selects the top-k compressed blocks per query. Result: 1M tokens at ~27% of the FLOPs and ~10% of the KV cache of V3.2.
• Heavily Compressed Attention (HCA): More aggressive compression (m' >> m), dense (no sparse selection); interleaved with CSA layers in the hybrid architecture. V4-Flash reaches 10% of the FLOPs and 7% of the KV at 1M.
• Lightning indexer in FP4: The indexer's QK queries run in FP4 (MXFP4) — index scores quantized from FP32 to BF16 give a 2× speedup on the top-k selector with 99.7% recall.
• Complementary attention: Sliding-window branch (n_win most recent uncompressed KVs) + attention sink with learnable logits + partial RoPE (only on the last 64 dims of the queriesKVsoutputs).
• Manifold-Constrained Hyper-Connections (mHC): Constrains the residual matrix to the manifold of doubly-stochastic matrices (Birkhoff polytope) via Sinkhorn-Knopp 20 iters; guarantees spectral norm ≤ 1 (non-expansive mapping), eliminating the numerical instability of conventional HC in deep stacks.
• Muon Optimizer: Replaces AdamW for most modules (AdamW persists in the embedding, prediction head, static biases, mHC gating, RMSNorm). Uses Hybrid Newton-Schulz: 8 iters with coefs (3.4445, −4.7750, 2.0315) for fast convergence + 2 iters with (2, −1.5, 0.5) to stabilize singular values at 1. No QK-Clip (RMSNorm on queries/KV is sufficient).
• MoE changed vs V3: activation function Sigmoid → Sqrt(Softplus); sequence-wise balance loss + auxiliary-loss-free; hash routing in the first layers (replaces the initial dense FFN); removed the constraint on the number of routing target nodes.
• MTP (Multi-Token Prediction): Kept identical to V3.
• Cross-tool reasoning: Keeps the full reasoning history between tool calls (V3.2 discarded it).
• Three reasoning modes: Non-Think / Think High / Think Max (via the <think> token).

Hybrid KV cache — inference engineering (§3.6):

• Heterogeneous KV cache: two components — a classical block cache (CSA Indexer KV + CSA Main KV + HCA KV, block size = lcm(m, m')) + a per-request state cache (SWA KV + uncompressed tail tokens not yet ready for compression).
• On-disk KV cache for shared-prefix reuse: eliminates repeated re-prefill in long shared prompts.
• Mixed-precision storage: RoPE dims in BF16, other dims in FP8 → ~50% savings vs pure BF16. This scheme, combined with CSA+HCA, brings the KV cache at 1M to ~2% of the BF16 GQA8 baseline.

Training (§3):

• FP4 Quantization-Aware Training (QAT): MXFP4 applied to (1) MoE expert weights and (2) the indexer's QK path in CSA. FP4-to-FP8 dequantization is lossless (E4M3 absorbs the scales of the 1×32 sub-blocks within 128×128 FP8 blocks). Real FP4 weights used in inference and RL rollout.
• Bitwise train↔inference determinism: separate accumulation buffers per SM in the attention backward; token-order pre-processing + buffer isolation in the MoE backward; split-k mHC with reduction in a separate kernel.
• Hybrid ZeRO for Muon (Muon needs the full gradient matrix — classic ZeRO assumes element-wise optimizers): knapsack for dense parameters, flattening of the MoE experts for uniform distribution; gradients synchronized in BF16 with stochastic rounding (–50% comm); reduce-scatter replaced by all-to-all + local FP32 sum.
• mHC overhead is only 6.7% of the wall-time of the 1F1B pipeline — thanks to fused kernels + selective recomputation + DualPipe tuning.
• TileLang DSL for kernel development (the substrate of all custom kernels).

Two-phase post-training:

1. SFT + GRPO per specialized domain
2. Consolidation via on-policy distillation

V4-Pro-Max benchmarks:

API prices (vs competitors):

• V4-Flash: \(0.14/M tokens (vs GPT-5-Nano: \)0.20)
• V4-Pro: \(1.74/M tokens (vs Claude Sonnet 4.6: \)3.00)

---

Mistral

Mistral Small 4 (03162026): Combines reasoning (Magistral), vision (Pixtral) and agentic coding (Devstral) in a single model. 128 experts with 6B active per token. Voxtral TTS (03232026): Mistral's first audio bet; open-weights; support: EN, FR, DE, ES, NL, PT, IT, HI, AR. Leanstral: First open-source agent for formal verification in Lean 4; 6B active; comes with FLTEval (evaluation suite). Links: mistral.ai · huggingface.co/mistralai

---

Gemma (Google)

DiffusionGemma (06102026): the family's first open-weights language diffusion model — generates blocks of 256 tokens in parallel instead of token-by-token autoregressive. Same Gemma 4 base (26B MoE / 4B active), multimodal (text+image+video→text), 18 GB VRAM quantized, MLXvLLMHFUnsloth integration + NVFP4. Trade-off: ~4× throughput at the cost of quality below the AR Gemma 4 — ideal for real-time infilleditingauto-correction. Architectural detail in [[..02-architectures/alternative-architectures]] § Language Diffusion Models.

Gemma 4 (04022026): Built on the same technology as Gemini 3. The first time the Gemma family uses Apache 2.0 across all sizes. Support for text, images, audio and code; 140+ languages.

Gemma 4 31B — Benchmarks:

Links: ai.google.devgemma · huggingface.cogoogle · deepmind.googlemodelsgemma

---

Phi (Microsoft)

Links: huggingface.comicrosoft · microsoft.comresearch

---

Kimi K2.6 (Moonshot AI)

Kimi K2.6 (04202026): 384 experts (8 selected + 1 shared), 61 layers, 64 attention heads, MLA, vision with MoonViT (400M params). The Agent Swarm system scales up to 300 sub-agents with 4,000 coordinated steps.

K2.6 benchmarks:

For Kode: Interesting for agentic coding in wide context; the license allows commercial use.

---

MiniMax M3 (MiniMax AI)

MiniMax M3 (06012026): the first open-weight model to combine frontier coding, 1M-token context and native multimodality in a single model. Text, image and video inputs → text output; operates desktop (computer use). Mixed-modality training from step 0, with a pipeline rebuilt for interleaved data (text+image in the same sequence), scaled to the order of 100T tokens. Parameter/expert counts not disclosed in the announcement; tech report + weights promised ~10 days after launch.

Architecture: Sparse MoE + MSA (MiniMax Sparse Attention) — sparse selection of KV blocks over uncompressed KV (contrast with DeepSeek's MLA and CSADSA, which compress). At 1M context: computetoken ≈ 120 of M2, prefill ~9.7× and decoding ~15.6× faster. Detail in `02-architecturestransformer-and-attention.md` (MSA section).

M3 benchmarks:

For Kode: a strong candidate for agentic coding + long context + multimodal, at a reported cost of ~5–10% of GPT-5.5Gemini 3.1 Pro. Note on the license: open-weight but with commercial use conditions (not pure MITApache) — review the terms before adopting. Not yet covered in Koder's local base (services/ai/runtime/models.yaml and the modelreg family enum do not include MiniMax; there is only a reference as a cloud provider in services/ai/ai/cli). A candidate for inclusion once the weights are published.

---

Nemotron 3 (NVIDIA)

Nemotron 3 Super (04032026) — Technical report: research.nvidia.comlabsnemotronfilesNVIDIA-Nemotron-3-Super-Technical-Report.pdf

LatentMoE: A new MoE architecture that projects expert weights into a shared latent space, reducing total parameters while maintaining capacity. Better accuracy per parameter and per FLOP than regular MoEs.

NVIDIA Nemotron Coalition: Black Forest Labs, Cursor, LangChain, Mistral AI, Perplexity, Reflection AI, Sarvam, Thinking Machines Lab — building Nemotron 4 together.

---

OLMo (Allen AI)

OLMo 3 — Paper: arXiv:2512.13961. Includes intermediate checkpoints, all data, dependencies. OLMo Hybrid (03052026): Combines attention layers (Transformer) with linear RNN layers. Achieves the same accuracy as OLMo 3 with 49% fewer tokens. Trained on NVIDIA H100 → B200. First SOTA model trained on B200s in production.

---

Command A (Cohere)

Paper: arXiv:2504.00698 (Cohere, Apr/2026). Own alignment algorithms: SRPO (Self-Rewarding Preference Optimization) and CoPG (Contrastive Preference Gradient). Multi-phase polish pipeline for enterprise delivery.

---

Other Relevant Models

---

Specialized Code Models

---

Specialized Code Models

---

Recommendation for Kode